← Master Index
Vol. 13 Module 13.3 Lecture

Token Budgeting

Token Management & Usage (added)

How This Lesson Fits the Module & Volume

After counting, token budgeting assigns hard slices of the context to system, tools, history, retrieved docs, and reserved output. Without a budget, any one component can starve the others.

This lesson operationalizes the Vol. 11 context window: the window is a shared pool, not an infinite notepad.

Related foundations: recount tokens with tiktoken (Vol. 12) and keep payloads inside the context window (Vol. 11).

Learning Objectives

By the end of this lesson, students should be able to:

  • Draw a token budget pie for a production chat/RAG call.
  • Reserve output (completion) tokens before packing inputs.
  • Prioritize mandatory vs optional context when over budget.
  • Express budgets as constants in config, not magic numbers in prompts.
  • Stress-test budgets with long histories and large tool schemas.
  • Connect budgets to later cost estimation in Module 13.4.
Definition

Token budgeting is allocating a fixed context window capacity across prompt components and a reserved completion allowance so no single part overflows the model limit.

A Practical Budget Pie

SliceTypical shareNotes
System + tools5–15%Stable; cache-friendly
Conversation history20–40%Truncate oldest first
Retrieved / attached docs30–50%Chunk-level planning later
Reserved output10–25%Protect max_tokens headroom

Code: Enforce a Budget

import tiktoken enc = tiktoken.encoding_for_model("gpt-4o") WINDOW = 128_000 RESERVE_OUT = 2_000 MAX_INPUT = WINDOW - RESERVE_OUT BUDGET = { "system": 1_500, "history": 8_000, "docs": MAX_INPUT - 1_500 - 8_000 - 500, # leave slack "user": 500, } def fit(text: str, limit: int) -> str: ids = enc.encode(text) if len(ids) <= limit: return text return enc.decode(ids[:limit]) # Pack each slice under its ceiling, then assert total <= MAX_INPUT.

Static budgets

  • Simple ops
  • Easy SLOs
  • May waste headroom

Dynamic budgets

  • Steal from history when docs grow
  • More code
  • Needs good priorities

Always

  • Reserve output first
  • Fail closed if over
  • Log slice sizes

Strengths

  • Predictable latency & quality
  • Clear ownership per slice
  • Pairs with cost caps

Tradeoffs

  • Rigid pies waste tokens
  • Tool schemas grow silently
  • Must revisit when window changes
Common Misconception

“We have a 128k window, so budgeting is optional.” Large windows still cost money, slow attention, and dilute focus. Budgets protect quality and spend even when the hard limit is far away.

Knowledge Check

  1. Short Answer: What should you reserve before packing inputs? Answer: Output/completion token headroom.
  2. True/False: A token budget is only needed for models under 8k context. Answer: False.
  3. Multiple Choice: Best first cut when over budget: (a) delete system prompt, (b) drop lowest-priority optional context, (c) raise temperature. Answer: (b).
  4. Short Answer: Name three common budget slices. Answer: System/tools, history, docs/retrieval, user, output reserve (any three).
  5. True/False: Budgets should live in config/code, not tribal knowledge. Answer: True.
  6. Multiple Choice: Vol. 11 concept for total capacity: (a) learning rate, (b) context window, (c) dropout. Answer: (b).
  7. Short Answer: Why log per-slice sizes? Answer: To diagnose which component blows the budget.
  8. Short Answer: How does budgeting help Module 13.4? Answer: Stable token volumes make cost forecasts reliable.
  9. Multiple Choice: Output reserve of 0 mainly risks: (a) cheaper calls, (b) truncated answers / hard failures, (c) better RAG. Answer: (b).
  10. True/False: Tool JSON schemas never consume budget. Answer: False.

Key Takeaways

  • Treat the window as a shared pie with an output reserve.
  • Encode budgets in config; measure each slice.
  • Drop optional context before mandatory instructions.
  • Large windows do not remove the need to budget.
  • Next: Context Window Management.
Trainer’s Guide

Lab: Given a 32k window, design a pie for a support bot with tools + RAG; defend the output reserve.

Discussion: When is stealing tokens from history worse than truncating docs?

Recap: Budgeting turns raw counts into enforceable allocations. Continue with Context Window Management.