Token Overflow Handling — Module 13.3

How This Lesson Fits the Module & Volume

Overflow is what happens when prevention fails: payload exceeds the {CTX} or account ceilings. Handling must be explicit—retry with trim, degrade gracefully, or fail closed with a clear error.

Closes the loop with limits, truncation, and budgeting before the final max_tokens control lesson.

Related foundations: recount tokens with tiktoken (Vol. 12) and keep payloads inside the context window (Vol. 11).

Learning Objectives

By the end of this lesson, students should be able to:

Detect overflow via errors, finish reasons, and preflight counts.
Implement retry-with-trim vs fail-closed policies.
Preserve critical instructions across remediation.
Surface actionable errors to clients and operators.
Avoid infinite retry loops on permanent overflows.
Capture metrics for overflow rate as an SLO signal.

Definition

Token overflow handling is the runtime response when a request would exceed or has exceeded token capacity—context, output, or throughput—including detection, remediation, and user communication.

Signal	Meaning	Handler
Preflight count > budget	Preventable	Trim / reject before call
API context_length error	Hard overflow	Shrink & retry once or fail
finish_reason=length	Hit max output	Continue turn or summarize
429 TPM	Rate overflow	Backoff (see 13.4)

Code: Fail Closed with One Trim Retry

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
MAX_IN = 120_000

def shrink(text: str, limit: int) -> str:
    ids = enc.encode(text)
    return enc.decode(ids[:limit]) if len(ids) > limit else text

def call_with_overflow_guard(prompt: str, call_fn):
    n = len(enc.encode(prompt))
    if n > MAX_IN:
        prompt = shrink(prompt, MAX_IN)
    try:
        return call_fn(prompt)
    except ContextLengthError:
        prompt2 = shrink(prompt, int(MAX_IN * 0.8))
        if prompt2 == prompt:
            raise
        return call_fn(prompt2)  # single retry

Fail closed

No silent loss
Clear UX
May block power users

Auto-trim retry

Higher success rate
Hidden deletions
Must log

Degrade mode

Smaller model / no RAG
Quality drop
Keeps availability

Strengths

Stops cryptic crashes
Protects spend and SLOs
Creates operable metrics

Tradeoffs

Retry storms if misconfigured
Trim may remove key evidence
Needs good client messaging

Common Misconception

“Catch all exceptions and resend the same payload.” Permanent context overflows will loop forever. Detect overflow class errors and change the payload or stop.

Knowledge Check

Short Answer: What is a preflight overflow check? Answer: Counting tokens before the API call against a budget.
True/False: finish_reason=length indicates output hit its cap. Answer: True.
Multiple Choice: Infinite retries on context errors are: (a) best practice, (b) dangerous, (c) required by HTTP. Answer: (b).
Short Answer: Name one remediation. Answer: Trim/truncate, summarize, drop RAG, or fail closed.
True/False: Overflow rate is a useful ops metric. Answer: True.
Multiple Choice: 429 TPM is primarily: (a) rate overflow, (b) a tokenizer, (c) an embedding. Answer: (a).
Short Answer: What must logs include on auto-trim? Answer: That truncation occurred and roughly what was dropped.
Short Answer: How does tiktoken help? Answer: Detect/prevent overflow before calling.
Multiple Choice: Safety text during trim should be: (a) discarded first, (b) preserved, (c) randomized. Answer: (b).
True/False: Overflow handling is only a client UI concern. Answer: False—server policy too.

Key Takeaways

Detect early; remediate once; fail clearly.
Never retry identical overflowing payloads.
Log trims; preserve critical spans.
Track overflow rate as an SLO.
Next: Max Tokens Parameter.

Trainer’s Guide

Lab: Inject oversized prompts in a staging harness; verify single-retry trim and alert.

Discussion: Write the user-facing error copy for overflow vs rate limit.

Recap: Overflow handling is the safety net under budgeting. Continue with Max Tokens Parameter.