Next-token prediction conditions on “everything so far”—but Transformers do not have infinite memory. The context window is the maximum number of tokens the model can attend over in one forward pass. It is set by architecture (positional encodings, attention cost) and by product config.
This limit shapes prompting, RAG chunking, chat memory, and why KV cache memory scales with sequence length during inference.
Learning Objectives
By the end of this lesson, students should be able to:
- Define context window as the max token length of visible context (prompt + generation so far).
- Explain why attention cost and KV cache grow with sequence length (roughly O(T²) compute, O(T) memory per layer for cache).
- Distinguish model max length from API “context” budgets that also reserve output tokens.
- Count tokens with a Hugging Face tokenizer and truncate or window long inputs safely.
- List practical strategies: sliding windows, summarization, RAG, long-context variants.
- Connect positional embeddings from Vol. 10 to length extrapolation limits.
The context window (context length, max sequence length) is the largest number of tokens a model can process as a single contiguous sequence. Tokens outside the window are invisible to attention unless brought back via retrieval, memory mechanisms, or a new call.
What Fits in the Window?
Instructions / tools schema.
Prior chat turns.
Current prompt / docs.
New tokens still count.
| Era / example | Typical window | Implication |
|---|---|---|
| GPT-2 | 1,024 | Short documents only |
| Early GPT-3 style | 2k–4k | Few-shot prompts compete with docs |
| Modern mid LLMs | 8k–32k | Multi-file chats become feasible |
| Long-context models | 100k+ | Still costly; quality can degrade |
Why There Is a Limit
Compute
- Dense attention ~ O(T²) per layer.
- Long prompts get expensive fast.
- Motivated sparse / linear attention research.
Memory
- Activations scale with T.
- KV cache stores keys/values per token.
- VRAM often binds batch × length.
Positions
- Trained positional range is finite.
- Extrapolation needs RoPE scaling, etc.
- Beyond train length ≠ free lunch.
Code: Count, Truncate, and Budget
Working Around the Window
Strategies
- RAG: retrieve only relevant chunks.
- Summarize / hierarchical memory.
- Sliding window over long files.
- Long-context fine-tunes / RoPE scale.
Failure Modes
- Silent truncation drops critical instructions.
- “Lost in the middle” on very long prompts.
- Counting characters instead of tokens.
“128k context means the model reliably uses all 128k tokens equally.” Capacity is not the same as effective use. Attention can underweight the middle of long prompts, and quality often peaks well below the advertised maximum. Always measure task performance at the lengths you actually ship—and budget tokens for the completion, not only the prompt.
Knowledge Check
- Short Answer: What unit is a context window measured in? Answer: Tokens (not characters or words).
- True/False: Generated tokens do not count against the context window. Answer: False—they append to the sequence and consume budget.
- Multiple Choice: Dense self-attention compute scales roughly: (a) O(T), (b) O(T log T), (c) O(T²). Answer: (c).
- Short Answer: Name one reason positional encodings limit length. Answer: The model was trained on a finite position range; extrapolating may degrade (any similar wording).
- True/False: KV cache memory grows with the number of cached tokens. Answer: True.
- Multiple Choice: A safe long-doc tactic is: (a) ignore the limit, (b) truncate/window or retrieve chunks, (c) lowercase everything. Answer: (b).
- Short Answer: Why reserve output tokens when packing a prompt? Answer: So generation does not immediately hit max length / get cut off.
- Short Answer: What is “lost in the middle”? Answer: Models often use early and late context better than middle spans in long prompts.
- Multiple Choice: Character length is: (a) identical to token length, (b) only a rough proxy for tokens, (c) always larger than tokens. Answer: (b).
- True/False: RAG can reduce how many tokens must sit in the window at once. Answer: True.
Key Takeaways
- The context window is the max tokens visible to the model in one sequence.
- Limits come from attention cost, memory (KV cache), and positional training range.
- Prompt + history + completion share one budget—count tokens, not characters.
- Mitigations: truncation policies, sliding windows, summarization, RAG, long-context models.
- Next: Vocabulary—the discrete set those tokens are drawn from.
Hands-on idea: Have students tokenize the same paragraph with GPT-2 vs a multilingual tokenizer and compare lengths.
Discussion prompt: Design a chat product with a 4k window: what do you keep when history overflows?
Recap: Context windows bound how much prefix an LM can condition on; plan prompts and memory around token budgets. Continue with Vocabulary.