← Master Index
Vol. 11 Module 11.1 Lecture

Language Model

Language Model Concepts

How This Lesson Fits the Module & Volume

Volume 10 closed with the Vision Transformer: the same stack of multi-head self-attention, feed-forward blocks, and residuals—applied to image patches. You also built decoder blocks with causal masked attention, the ingredient that lets a model generate left-to-right.

Volume 11 starts here: a language model (LM) is that decoder stack used as a next-token probability machine. Module 11.1 unpacks every supporting idea—probability distributions, next-token prediction, tokens, tokenizers, embeddings, logits, sampling, and inference—so you can read modern LLMs as engineering, not magic.

Learning Objectives

By the end of this lesson, students should be able to:

  • Define a language model as a conditional distribution over the next token given context.
  • Connect Vol. 10 decoder stacks (causal masking) to autoregressive LM generation.
  • Contrast encoder-only, decoder-only, and encoder–decoder Transformers for language tasks.
  • Sketch the LM forward path: tokenize → embed → Transformer → logits → softmax.
  • Run a tiny Hugging Face causal LM and inspect next-token probabilities.
  • Map Module 11.1 topics onto pieces of that pipeline.
Definition

A language model assigns a probability to sequences of tokens. In the autoregressive (causal) form used by GPT-style models, it factors as a product of next-token distributions: P(x1…xT) = ∏t P(xt | x<t). At each step the network outputs a distribution over the vocabulary.

From ViT Patches to Text Tokens

ViT proved Transformers are modality-agnostic: tokens can be patches or subwords. Language modeling flips the objective. Instead of classifying a whole sequence, a causal LM predicts the next discrete symbol given everything so far. The architecture is essentially the Vol. 10 decoder stack—self-attention with a causal mask so position t never sees the future.

1. Tokenize

Text → token IDs (tokenizer).

2. Embed

IDs → vectors (embedding).

3. Decode

Causal Transformer stack.

4. Predict

Logits → softmax over vocab.

Three Transformer Layouts for Language

LayoutAttentionTypical useExample
Encoder-onlyBidirectionalUnderstanding / classificationBERT (Module 11.2)
Decoder-onlyCausal (masked)Generation / chatGPT, Llama
Encoder–decoderFull + cross-attnSeq2seq (translate, summarize)T5, BART

What Vol. 10 Gave You

  • Attention, Q/K/V, MHSA.
  • Encoder vs decoder blocks.
  • Causal masking for generation.
  • Positional encodings / embeddings.

What Vol. 11 Adds

  • LM as next-token probability.
  • Vocab, tokens, tokenizers at scale.
  • Logits, temperature, top-k / top-p.
  • KV cache and inference practice.

Training Signal

  • Cross-entropy on next token.
  • Teacher forcing on full sequences.
  • Same backprop stack as Vol. 06.
  • Scale: data + params + compute.

A Minimal Causal LM Call (Hugging Face)

This loads a small GPT-2, scores the next-token distribution after a prompt, and prints the top candidates. You will refine each piece—distribution, sampling, tokens—in later lectures.

import torch from transformers import AutoModelForCausalLM, AutoTokenizer name = "gpt2" tok = AutoTokenizer.from_pretrained(name) model = AutoModelForCausalLM.from_pretrained(name) model.eval() prompt = "Language models predict" ids = tok(prompt, return_tensors="pt") with torch.no_grad(): out = model(**ids) logits = out.logits[0, -1] # last position probs = torch.softmax(logits, dim=-1) top = torch.topk(probs, k=5) for p, i in zip(top.values.tolist(), top.indices.tolist()): print(f"{p:.4f} {tok.decode([i])!r}")

Strengths and Tradeoffs

Strengths

  • One objective (next token) scales to many tasks via prompting.
  • Reuses Vol. 10 architecture with a clear probabilistic story.
  • Open tooling (PyTorch, Hugging Face) for research and product.

Tradeoffs

  • Autoregressive decoding is sequential and can be slow.
  • Finite context window limits long documents.
  • Probabilities ≠ truth; hallucinations remain a systems problem.
Common Misconception

“A language model understands text the way a person does.” Operationally, an autoregressive LM is a highly capable conditional next-token predictor. Fluency and world knowledge emerge from that objective at scale—but the model still outputs a distribution over tokens, not a verified belief. Treat outputs as samples from that distribution until grounded by tools or retrieval.

Module 11.1 Roadmap

After this overview you will deepen the probability story, then the discrete units (vocab / tokens / tokenizer), then continuous representations (embedding / embedding space), then the inference stack (hidden state, logits, softmax, sampling, temperature, top-k/p, beam search, KV cache).

Knowledge Check

  1. Short Answer: Write the autoregressive factorization of a sequence probability. Answer: P(x1…xT) = ∏t P(xt | x<t).
  2. True/False: GPT-style LMs use bidirectional attention like BERT. Answer: False—they use causal (masked) self-attention.
  3. Multiple Choice: ViT and GPT both: (a) classify images, (b) treat inputs as token sequences through a Transformer, (c) require CNNs. Answer: (b).
  4. Short Answer: Name the four high-level LM pipeline stages. Answer: Tokenize, embed, Transformer decode, logits/softmax over vocab.
  5. True/False: Encoder-only models are the default choice for open-ended chat generation. Answer: False—decoder-only (or encoder–decoder) models are used for generation.
  6. Multiple Choice: The training loss for a causal LM is typically: (a) MSE on embeddings, (b) cross-entropy on the next-token distribution, (c) hinge loss. Answer: (b).
  7. Short Answer: Why does causal masking matter for language modeling? Answer: It prevents attending to future tokens so next-token prediction is a valid left-to-right task.
  8. Short Answer: What does out.logits[0, -1] mean in the HF snippet? Answer: Logits for the last sequence position of batch item 0—the next-token scores.
  9. Multiple Choice: A finite context window means: (a) infinite memory, (b) only the last N tokens are visible, (c) vocab size is N. Answer: (b).
  10. True/False: High next-token probability guarantees factual correctness. Answer: False.

Key Takeaways

  • A modern LM is a Transformer (usually decoder-only) that outputs P(next token | context).
  • Vol. 10’s causal decoder stack is the architectural core; Vol. 11 adds the LM objective and tooling.
  • Encoder-only, decoder-only, and encoder–decoder layouts serve different language tasks.
  • The pipeline is tokenize → embed → stack → vocab logits → distribution / sample.
  • Next: Probability Distribution—what the model’s output vector means.
Trainer’s Guide

Hands-on idea: Run the GPT-2 snippet on three prompts and have students predict the top token before revealing probabilities.

Discussion prompt: “If ViT and GPT share the same attention math, what actually makes one a vision classifier and the other a language model?” (Answer: input representation + objective + masking.)

Recap: Language models turn Vol. 10 decoder stacks into next-token probability engines. Continue with Probability Distribution.