Earlier you studied the context window as a systems concept. Context length is the engineering number that bounds how many tokens an LLM can attend over in one forward pass. It shapes RAG chunking, agent memory, cost, and quality—and closes Module 11.4 before Module 11.5’s model families advertise ever-longer windows.
Learning Objectives
By the end of this lesson, students should be able to:
- Define context length / context window in tokens.
- Explain compute/memory scaling with sequence length.
- Relate positional methods (RoPE scaling, ALiBi, etc.) to long context.
- Identify “lost in the middle” and needle-in-haystack eval ideas.
- Choose chunking/RAG strategies when documents exceed the window.
- Trade off longer context vs retrieval quality and cost.
Context length is the maximum number of tokens a model can accept as input (and, together with generated tokens, often as total sequence) while applying attention. It is a hard architectural and serving limit, not a soft preference.
Why Length Is Expensive
Dense self-attention scales roughly O(n²) in sequence length for compute and KV cache memory. Longer windows enable better in-context learning and whole-document reasoning, but raise latency, GPU memory, and dollar cost per request.
| Strategy | Idea | Tradeoff |
|---|---|---|
| Longer native window | Train/finetune for large n | Costly pretrain & serve |
| Positional extrapolation | RoPE scaling, YaRN, ALiBi | Quality may degrade past train length |
| RAG / hierarchical summarize | Fetch only needed spans | Retrieval errors; extra pipeline |
| Sliding / map-reduce | Process chunks, merge | Cross-chunk reasoning harder |
Effective vs Advertised Context
Advertised
- Max tokens the API accepts.
- Marketing headline number.
Effective
- Where accuracy stays high.
- Needle tests; mid-context drop.
Does the task fit in n tokens?
Put best evidence near ends / structured.
Retrieve, summarize, or multi-call.
Cost, latency, faithfulness.
Long Context Wins
- Whole-file coding edits.
- Multi-doc synthesis in one shot.
- Fewer retrieval round-trips.
Long Context Costs
- Quadratic-ish attention bills.
- Diluted attention / lost-in-middle.
- Easier to stuff noisy context.
“A 1M-token window means the model equally uses every token.” Utilization is uneven; performance often peaks when critical facts are easy to attend to. Measure effective context, don’t worship the max.
Knowledge Check
- Short Answer: What is context length? Answer: The max tokens a model can condition on in one sequence/window.
- True/False: Dense attention cost grows linearly with length always. Answer: False—standard dense attention is roughly quadratic.
- Multiple Choice: When a doc exceeds the window, a common approach is: (a) RAG/chunking, (b) deleting the model, (c) only raising batch size, (d) removing the tokenizer. Answer: (a).
- Short Answer: What is “lost in the middle”? Answer: Performance drop for information placed mid-context versus edges.
- True/False: Advertised max context always equals effective usable context. Answer: False.
- Multiple Choice: KV cache pressure grows with: (a) shorter prompts only, (b) longer sequences / generations, (c) font size, (d) CSS themes. Answer: (b).
- Short Answer: Name one positional technique related to long context. Answer: RoPE scaling, YaRN, ALiBi, etc.
- True/False: Stuffing irrelevant tokens into a long window can hurt quality. Answer: True.
- Multiple Choice: Module 11.5 begins with surveying: (a) CNN pools only, (b) modern LLM families, (c) only SVMs, (d) only k-means. Answer: (b).
- Short Answer: Give one reason to prefer RAG over maxing context. Answer: Lower cost, fresher facts, and focused evidence beats noisy full dumps.
Key Takeaways
- Context length caps how much text an LLM can see at once.
- Longer windows help but cost compute/memory and may not be fully used.
- RAG and hierarchical strategies complement raw window growth.
- Next module: GPT Family and other modern LLM lineages.
Needle demo: Hide a fact at start/middle/end of a long prompt; compare recovery rates.
Budgeting: Have students allocate a 8k window among system, retrieved chunks, and user turns.
Recap: Context length is a hard token budget with cost and quality implications. Continue to Module 11.5 with GPT Family.