Max Tokens Parameter — Module 13.3

How This Lesson Fits the Module & Volume

The max_tokens (or max_completion_tokens) parameter is the primary output faucet: it caps generation, reserves window headroom, and bounds worst-case cost for a call.

This lecture caps Module 13.3 and hands off to Module 13.4 pricing—where output caps meet dollar rates.

Related foundations: recount tokens with tiktoken (Vol. 12) and keep payloads inside the context window (Vol. 11).

Learning Objectives

By the end of this lesson, students should be able to:

State what max_tokens does and does not control.
Choose values from task needs + cost ceilings.
Combine with stop sequences and format instructions.
Leave context headroom: input + max_tokens ≤ window.
Handle finish_reason=length with continuations or redesign.
Differentiate provider parameter names across APIs.

Definition

max_tokens (names vary by API) is the upper bound on how many new tokens the model may generate in a completion. It does not by itself shrink the prompt.

Setting	Effect	Typical use
Low (64–256)	Short answers; cheap	Classification, tool args
Medium (512–2k)	Paragraphs / JSON	Support replies
High (4k+)	Long form / code	Needs strong cost controls
Unset / huge	Model/SKU default risk	Avoid in prod without budget

Code: Headroom Check

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
WINDOW = 128_000

def validate_request(prompt: str, max_tokens: int) -> dict:
    prompt_tokens = len(enc.encode(prompt))
    if prompt_tokens + max_tokens > WINDOW:
        raise ValueError("input + max_tokens exceeds context window")
    # example asymmetric pricing for planning
    in_rate, out_rate = 2.5 / 1e6, 10.0 / 1e6
    worst_cost = prompt_tokens * in_rate + max_tokens * out_rate
    return {"prompt_tokens": prompt_tokens, "worst_case_usd": worst_cost}

print(validate_request("Write a haiku about TPM.", max_tokens=64))

max_tokens

Hard generation cap
Cost/latency bound
May cut mid-thought

Stop sequences

Semantic end
May never fire
Pairs with cap

Prompt brevity asks

Soft control
Model may ignore
Still set a hard cap

Strengths

Bounds worst-case output spend
Protects context headroom
Simple lever for SLOs

Tradeoffs

Too low truncates useful answers
Name differs across vendors
Easy to forget in SDKs

Common Misconception

“Setting max_tokens=128000 lets me send a 128000-token prompt.” That value caps output. Prompt size is separate; together they must fit the context window.

Knowledge Check

Short Answer: What does max_tokens cap? Answer: Generated/completion tokens (output).
True/False: max_tokens alone truncates the prompt. Answer: False.
Multiple Choice: input + max_tokens should be: (a) ≤ context window, (b) unlimited, (c) equal to temperature. Answer: (a).
Short Answer: What finish_reason often means the cap was hit? Answer: length (or equivalent).
True/False: Low max_tokens can reduce cost. Answer: True.
Multiple Choice: Classification endpoints often use: (a) very high caps, (b) low caps, (c) no tokenizer. Answer: (b).
Short Answer: Why compute worst-case USD with max_tokens? Answer: Output may run to the cap; plan for the ceiling.
Short Answer: Name a soft complement to max_tokens. Answer: Stop sequences or concise-format instructions.
Multiple Choice: Module next for $/token: (a) 13.4 pricing, (b) Vol. 5 PCA, (c) CSS Grid. Answer: (a).
True/False: Leaving max_tokens unset is ideal for all production traffic. Answer: False.

Key Takeaways

max_tokens is the output ceiling.
Always check input + cap vs window.
Use it for cost worst-case planning.
Handle length finishes deliberately.
Next module: 13.4 Per-Token Pricing.

Trainer’s Guide

Lab: For three task types, pick max_tokens and justify with worst-case cost at sample rates.

Discussion: How do you productize “continue generation” after finish_reason=length?

Recap: Max tokens closes Module 13.3’s control set. Continue into Per-Token Pricing.