Token Limits — Module 13.3

How This Lesson Fits the Module & Volume

Token limits are the hard ceilings on the context window, per-request size, and sometimes per-minute throughput. Hitting them without a plan means errors, silent cuts, or refused jobs.

This lecture catalogs limit types so later truncation, overflow handling, and rate limiting have a shared vocabulary.

Learning Objectives

By the end of this lesson, students should be able to:

Distinguish context, completion, TPM/RPM, and account quota limits.
Read model docs for max input/output before shipping.
Fail gracefully when approaching ceilings.
Validate planned payloads with tiktoken against documented maxima.
Separate provider hard limits from app-imposed caps.
Document limit assumptions in runbooks.

Definition

Token limits are maximum allowed tokens for a scope—per request context, per completion, per minute (TPM), or per billing period—enforced by the model, API gateway, or your own policy.

Limit type	Example	Failure mode
Context / max sequence	128k window	Request rejected or truncated
Max output	4k completion cap	Stop mid-answer
TPM / RPM	Tokens or requests per min	429 rate limit
Account quota	Monthly allowance	Hard block until reset/upgrade

Model limit

Architecture + product SKU
Same for all tenants of that model
Check model card

Account limit

Tier / spend plan
Varies by org
Dashboard + support

App limit

Your SLO
Can be stricter
Config-owned

Code: Preflight Against a Cap

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")

MODEL_CONTEXT = 128_000
MAX_OUT = 4_096

def preflight(prompt: str, want_out: int) -> None:
    n = len(enc.encode(prompt))
    if want_out > MAX_OUT:
        raise ValueError("want_out exceeds model max output")
    if n + want_out > MODEL_CONTEXT:
        raise ValueError(f"prompt {n} + out {want_out} > context {MODEL_CONTEXT}")

preflight("Hello " * 100, want_out=512)

Strengths

Predictable guardrails
Forces capacity planning
Aligns eng with billing tiers

Tradeoffs

Docs change; pins go stale
Multiple overlapping ceilings
Silent truncation if misconfigured

Common Misconception

“If the call returns 200, we were under all limits.” Some stacks truncate inputs silently or stop generation early. Always inspect finish reasons and usage.

Knowledge Check

Short Answer: Name two kinds of token limits. Answer: Context/max sequence and max output (or TPM/quota).
True/False: App-imposed caps may be lower than the model maximum. Answer: True.
Multiple Choice: TPM stands for: (a) Tokens Per Minute, (b) Tensor Product Model, (c) Total Prompt Memory. Answer: (a).
Short Answer: What HTTP status often signals rate limits? Answer: 429.
True/False: A 200 response guarantees no truncation. Answer: False.
Multiple Choice: Best preflight tool for OpenAI text: (a) tiktoken, (b) bathroom scale, (c) CSS rem. Answer: (a).
Short Answer: Why document limit assumptions? Answer: Model SKUs and account tiers change; runbooks need the pin.
Short Answer: How do limits relate to the context window? Answer: Context limit is the window ceiling for one call.
Multiple Choice: Monthly allowance is a: (a) positional encoding, (b) account quota, (c) attention head. Answer: (b).
True/False: Max output and context limit are the same number. Answer: False.

Key Takeaways

Know every ceiling: context, output, TPM, quota.
Preflight with the right tokenizer.
Inspect finish reasons, not only HTTP status.
Pin limits in docs/config.
Next: Truncation Strategies.

Trainer’s Guide

Lab: Map limits for two model SKUs your team uses; note which are account vs model.

Discussion: Should product hard-fail or auto-truncate when over limit? Tradeoffs?

Recap: Limits are the walls around counting and budgeting. Continue with Truncation Strategies.