Context Window Management — Module 13.3

How This Lesson Fits the Module & Volume

Vol. 11 defined the context window; this lecture manages it in live products—packing, sliding, summarizing, and retrieving so the model always sees the right tokens.

Module 13.3 sits between prompting craft (13.1–13.2) and cost control (13.4): window discipline is both a quality and a spend lever.

Learning Objectives

By the end of this lesson, students should be able to:

Restate the context window as prompt + generation capacity.
Apply sliding-window, summarization, and RAG packing strategies.
Separate hard model limits from soft product budgets.
Monitor effective context use (not just max length).
Handle multi-turn growth without unbounded history.
Link management tactics to tiktoken-based measurement.

Definition

Context window management is the set of policies that keep prompt + completion within the model’s context window while preserving task-critical information.

Management Playbook

Strategy	When	Watch-outs
Sliding window	Chat history growth	Loses early constraints
Running summary	Long threads	Summary drift / omissions
RAG retrieve-on-demand	Large corpora	Retrieval misses
Hard reject / ask user	Oversized uploads	UX friction

Hard limit

Provider/model max
Errors if exceeded
Non-negotiable

Soft budget

Product SLO / cost
May be tighter
Your policy

Effective use

% of window filled
Diminishing returns
Measure quality vs size

Code: Sliding History by Tokens

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")

def trim_history(messages, token_budget: int):
    """Keep newest messages that fit budget (role+content approx)."""
    kept = []
    used = 0
    for msg in reversed(messages):
        n = len(enc.encode(msg["content"])) + 4  # framing fudge
        if used + n > token_budget:
            break
        kept.append(msg)
        used += n
    return list(reversed(kept))

Strengths

Prevents overflow crashes
Keeps latency predictable
Forces explicit memory policy

Tradeoffs

Summaries can hallucinate
Sliding may drop safety rules
RAG adds infra complexity

Common Misconception

“Long-context models mean we can dump entire repos every turn.” Capacity ≠ attention quality or cost efficiency. Manage the window even at 100k+ tokens.

Knowledge Check

Short Answer: What does the context window bound? Answer: Tokens visible in one forward pass / API call (prompt + generation so far).
True/False: Soft product budgets can be stricter than the model max. Answer: True.
Multiple Choice: Best for huge knowledge bases: (a) paste all docs, (b) RAG on demand, (c) raise temperature. Answer: (b).
Short Answer: Name a risk of sliding-window chat trim. Answer: Dropping early system constraints or critical facts.
True/False: Filling 100% of a long window always improves answers. Answer: False.
Multiple Choice: Running summaries mainly risk: (a) free tokens, (b) drift/omissions, (c) GPU underuse. Answer: (b).
Short Answer: Why measure with tiktoken? Answer: Window policies must use the model’s token units.
Short Answer: Soft vs hard limit in one phrase. Answer: Soft = product/cost policy; hard = model/API ceiling.
Multiple Choice: Unbounded history growth causes: (a) inevitable overflow/cost, (b) zero tokens, (c) perfect recall forever. Answer: (a).
True/False: Context management is only a Vol. 11 theory topic. Answer: False—it is production policy here.

Key Takeaways

Manage windows with policy, not hope.
Combine trim, summarize, retrieve, and reject.
Soft budgets protect cost and focus.
Long context still needs discipline.
Next: Input Tokens vs Output Tokens.

Trainer’s Guide

Lab: Simulate a 50-turn chat; implement trim_history and compare answer quality vs full dump (until failure).

Discussion: Where should safety-critical rules live so sliding windows cannot erase them?

Recap: Window management keeps the Vol. 11 context window usable in production. Continue with Input vs Output Tokens.