Autoregressive Model — Module 11.3

How This Lesson Fits the Module & Volume

Decoder-only is the architecture; autoregressive is the probabilistic contract: the joint distribution over tokens factors left-to-right. This lecture ties Module 11.1 next-token prediction to GPT training and generation, and sets up causal attention as the mechanism that enforces the factorization.

Learning Objectives

By the end of this lesson, students should be able to:

Write the autoregressive factorization P(x1..xT)=Π P(xt | x<t).
Connect teacher-forced training to next-token cross-entropy.
Describe ancestral sampling at inference time.
Contrast AR LMs with masked (BERT) and diffusion-style generators at a high level.
Implement a tiny AR loss on shifted logits.
Relate temperature / top-p (Module 11.1) to AR decoding.

Definition

An autoregressive model expresses a sequence’s joint probability as a product of conditional distributions, each depending only on previous elements. In language modeling, each token is predicted from the prefix before it.

Factorization

For tokens x_1, …, x_T:

P(x) = P(x_1) P(x_2|x_1) … P(x_T|x_1..x_(T-1))

Training maximizes the likelihood of real text under this product (usually via mean token NLL). Generation draws tokens sequentially from each conditional—ancestral sampling.

Prefix

Context tokens so far.

Predict

Softmax over vocabulary.

Sample / argmax

Choose next id.

Append

Grow the prefix.

Mode	How conditionals are used
Teacher forcing (train)	Condition on gold previous tokens
Free-running (generate)	Condition on model’s own earlier samples
Teacher forcing benefit	Stable parallel training over positions
Exposure bias	Train/test mismatch from own errors

import torch
import torch.nn.functional as F

# logits: (B, T, V) predictions for positions 1..T given prefix
# targets: (B, T) gold tokens at those positions
def ar_nll(logits, targets):
    return F.cross_entropy(
        logits.reshape(-1, logits.size(-1)),
        targets.reshape(-1),
    )

B, T, V = 2, 5, 50
logits = torch.randn(B, T, V)
targets = torch.randint(0, V, (B, T))
print(ar_nll(logits, targets))

Autoregressive LM

Left-to-right factors
GPT training/inference
Streaming friendly

Masked LM

Predict blanks
Bidirectional context
Not ancestral text sampling

Why AR dominates chat

Natural token streaming
Simple likelihood
Scales with decoders

Common Misconception

“Autoregressive means the model cannot look at the whole user prompt.” It can attend to the entire prompt prefix; it simply cannot peek at future tokens that have not been generated yet.

Strengths and Tradeoffs

Strengths

Principled likelihood training.
Flexible controlled decoding.
Matches chat token streaming.

Tradeoffs

Sequential generation latency.
Error compounding in long samples.
Left-to-right bias on some tasks.

Knowledge Check

Short Answer: Write the AR factorization in words. Answer: Joint = product of each token given its past.
True/False: Teacher forcing conditions on gold prefixes during training. Answer: True.
Multiple Choice: Ancestral sampling: (a) draws tokens sequentially from conditionals, (b) only shuffles batches, (c) trains k-NN. Answer: (a).
Short Answer: What loss is typical for AR LMs? Answer: Cross-entropy / negative log-likelihood on next tokens.
True/False: MLM is the same factorization as AR LM. Answer: False.
Multiple Choice: Exposure bias refers to: (a) train on gold vs. generate on own tokens, (b) GPU heat only, (c) CSS. Answer: (a).
Short Answer: How does causal attention support AR? Answer: It blocks attending to future positions.
Short Answer: Name one Module 11.1 decoding knob. Answer: Temperature, top-k, or top-p.
Multiple Choice: AR generation is: (a) inherently sequential in tokens, (b) always one forward for infinite text, (c) convolution-only. Answer: (a).
True/False: The prompt tokens are part of the conditioning prefix. Answer: True.

Key Takeaways

Autoregressive LMs factor sequences left-to-right.
Train with teacher-forced next-token NLL; generate by sampling.
Causal masks enforce the conditional independence structure.
Distinct from bidirectional MLM objectives.
Next: Causal Attention.

Trainer’s Guide

Hands-on idea: Manually compute the product of three toy conditionals for a 3-token string.

Discussion prompt: How do beam search and sampling trade diversity vs. likelihood?

Recap: Autoregression is the GPT probability story. Continue with Causal Attention.