Token Optimization — Module 13.3

How This Lesson Fits the Module & Volume

Optimization reduces tokens without wrecking task success—shorter prompts, denser formats, fewer redundant examples, smarter retrieval. It bridges 13.3 management and 13.4 cost control.

You still measure with tiktoken; you just send fewer tokens for the same outcome.

Related foundations: recount tokens with tiktoken (Vol. 12) and keep payloads inside the context window (Vol. 11).

Learning Objectives

By the end of this lesson, students should be able to:

Identify high-ROI places to shrink prompts.
Prefer compact structured formats over verbose prose when safe.
Deduplicate history and retrieved chunks.
Trade few-shot count vs accuracy with measurements.
Avoid premature micro-optimizations that hurt clarity.
Quantify savings in tokens and projected dollars.

Definition

Token optimization is systematically reducing billed or budgeted tokens per successful task via content, format, caching, and routing changes—not by silently deleting critical instructions.

Lever	Example	Risk
Instruction compression	Bullet rules vs essays	Ambiguity
Example pruning	2-shot instead of 8-shot	Quality drop
Retrieval top-k	k=3 vs k=20	Missed evidence
Output schema	JSON fields vs narrative	User-facing tone
Caching / reuse	Stable system prefix	Stale policy

Code: Measure Before/After

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")

def ntok(s: str) -> int:
    return len(enc.encode(s))

verbose = """Please carefully read the following document and then
provide a comprehensive summary that covers all major points..."""
tight = "Summarize the document in 5 bullets. Cover risks and deadlines."

print(ntok(verbose), "->", ntok(tight), "saved", ntok(verbose) - ntok(tight))

Optimize input

Shorter system/RAG
Lower prefill cost
Watch quality gates

Optimize output

Strict formats, caps
Lower decode cost
May need post-process

Optimize routing

Small model when easy
Big savings
Needs classifiers

Strengths

Direct cost & latency wins
Fits more signal in the window
Encourages crisp prompts

Tradeoffs

Over-compression hurts clarity
Requires eval harness
Diminishing returns on tiny prompts

Common Misconception

“Delete the system prompt to save tokens.” That often increases retries and unsafe outputs—net tokens and risk go up. Optimize redundancy and retrieval first.

Knowledge Check

Short Answer: What is token optimization optimizing for? Answer: Fewer tokens per successful task (cost/latency/window).
True/False: You should measure token deltas with the model encoding. Answer: True.
Multiple Choice: High-ROI lever for huge corpora: (a) paste more, (b) better retrieval top-k, (c) raise max_tokens only. Answer: (b).
Short Answer: Name a risk of pruning few-shot examples. Answer: Quality/regression on edge cases.
True/False: Removing safety instructions is a recommended optimization. Answer: False.
Multiple Choice: Compact JSON output mainly reduces: (a) output tokens, (b) GPU VRAM forever, (c) dataset labels. Answer: (a).
Short Answer: Why A/B token savings with evals? Answer: Ensure quality does not collapse.
Short Answer: Which Vol. 12 tool measures the delta? Answer: tiktoken.
Multiple Choice: Deduplicating retrieved chunks primarily saves: (a) input tokens, (b) JPEG size, (c) CSS. Answer: (a).
True/False: Optimization ends once the prompt looks short to a human. Answer: False—verify in tokens.

Key Takeaways

Optimize with measurement and evals.
Prefer retrieval and dedupe over gutting instructions.
Compress both input and output sides.
Route easy work to cheaper models later (13.4).
Next: Chunk-Level Token Planning.

Trainer’s Guide

Lab: Take a verbose system prompt; produce a tight version; run 20-eval cases; report token and quality deltas.

Discussion: Where is optimization unethical (e.g., hiding required disclosures)?

Recap: Optimization is fewer tokens for the same success rate. Continue with Chunk-Level Token Planning.