← Master Index
Vol. 11 Module 11.3 Lecture

Causal Attention

GPT Family

How This Lesson Fits the Module & Volume

Autoregressive factorization needs an implementation mechanism: causal attention (masked self-attention). Volume 10 covered masked attention; here we specialize it for GPT-style LMs and connect to KV-cache-friendly inference from Module 11.1.

Learning Objectives

By the end of this lesson, students should be able to:

  • Define causal attention as forbidding attend-to-future positions.
  • Build / visualize a causal mask matrix.
  • Explain why the mask is required for parallel teacher-forced training.
  • Relate causal attention to streaming generation and KV caching.
  • Contrast causal vs. bidirectional attention with BERT.
  • Apply additive -inf masking before softmax in code.
Definition

Causal attention (look-ahead masked self-attention) restricts each query position i so it may attend only to key positions j ≤ i. Future tokens receive no attention weight, preserving autoregressive consistency.

Mask Shape

For sequence length T, a common Boolean upper-triangular mask marks forbidden (i, j) pairs with j > i. Implementations add large negative values to those logits before softmax so their probabilities become ~0.

k0k1k2k3
q0OKBLOCKBLOCKBLOCK
q1OKOKBLOCKBLOCK
q2OKOKOKBLOCK
q3OKOKOKOK

Bidirectional

  • All positions visible
  • BERT / MLM encoders
  • Not for next-token LM

Causal

  • Past + present only
  • GPT decoders
  • AR-safe

Prefix-LM hybrids

  • Bidirectional prompt
  • Causal on continuation
  • Some seq2seq recipes
import torch import torch.nn.functional as F def causal_attn(q, k, v): # q,k,v: (B, H, T, D) scores = (q @ k.transpose(-2, -1)) / (q.size(-1) ** 0.5) T = scores.size(-1) mask = torch.triu(torch.ones(T, T, dtype=torch.bool, device=q.device), diagonal=1) scores = scores.masked_fill(mask, float("-inf")) weights = F.softmax(scores, dim=-1) return weights @ v B, H, T, D = 1, 2, 4, 8 q = k = v = torch.randn(B, H, T, D) print(causal_attn(q, k, v).shape)

Training vs. Inference

Train

Full sequence + causal mask in parallel.

Infer step

Attend to cached past K/V.

Append

New token extends cache.

The mask guarantees that even when all tokens are processed in one training forward pass, no position peeks ahead—matching what generation will do one token at a time.

Common Misconception

“Causal attention means the model ignores the user prompt after the first token.” Prompt tokens are in the past of later positions; every new token can attend to the entire prompt and prior outputs.

Strengths and Tradeoffs

Strengths

  • Enables parallel AR training.
  • Matches streaming generation.
  • KV cache friendly.

Tradeoffs

  • No future context for disambiguation.
  • Still O(T²) without sparse/approx methods.
  • Mask bugs silently break AR validity.

Knowledge Check

  1. Short Answer: Position i may attend to which keys? Answer: Positions j ≤ i.
  2. True/False: Causal masks are unnecessary if you train left-to-right one token per step only. Answer: True for that scheme—but parallel teacher forcing needs the mask.
  3. Multiple Choice: Before softmax, blocked positions get: (a) -inf-like values, (b) +inf always, (c) dropout only. Answer: (a).
  4. Short Answer: Why use a causal mask during a full-sequence forward? Answer: To prevent leakage from future tokens while training in parallel.
  5. True/False: BERT uses the same causal mask as GPT. Answer: False.
  6. Multiple Choice: KV caching helps: (a) reuse past keys/values at decode time, (b) delete attention, (c) only train CNNs. Answer: (a).
  7. Short Answer: What does the upper triangle of a causal mask typically represent? Answer: Forbidden future attentions.
  8. Short Answer: Name the Volume 10 lecture on masked attention. Answer: Masked Attention.
  9. Multiple Choice: Softmax after -inf masking yields: (a) ~0 weight on blocked keys, (b) uniform future mass, (c) NaN always. Answer: (a).
  10. True/False: Causal attention alone chooses the next token string. Answer: False—the LM head + decoding policy do.

Key Takeaways

  • Causal attention blocks future positions for AR safety.
  • Masks enable parallel teacher-forced training.
  • Generation reuses the same constraint with KV caches.
  • Opposite of BERT’s bidirectional self-attention.
  • Next: Prompt.
Trainer’s Guide

Hands-on idea: Intentionally remove the mask in a tiny model and show training loss collapsing via future leak (toy demo).

Discussion prompt: How do prefix-LM hybrids change the mask pattern?

Recap: Causal attention is the AR enforcement layer. Continue with Prompt.