Token Usage — Module 19.1

How This Lesson Fits the Module & Volume

Latency asked how long a call takes. Token usage asks what you consumed and emitted—the quantity that drives both context-window pressure and dollars. In Vol. 19 it is an evaluation metric: two systems with the same accuracy can differ 10$\times$ in tokens (and cost).

This lecture is the eval-side companion to Vol. 13 token management: token counting, input vs output tokens, budgeting, per-token pricing, I/O pricing tiers, and cost estimation. Vol. 12 tiktoken is how you count locally; vendor usage objects are how you bill. Next: hallucination tests—quality, not spend.

Learning Objectives

By the end of this lesson, students should be able to:

Separate input (prompt) vs output (completion) tokens in usage reports.
Explain why output tokens usually cost more per token than input.
Turn usage into estimated USD with a price table (Vol. 13).
Treat tokens/request and tokens/successful-task as eval metrics.
Account for cache hits, reasoning/hidden tokens, and tool traces when exposed.
Compare systems fairly only under the same tokenizer and task success bar.

Definition

Token usage is the measured count of tokens a model consumes and generates for a request or a workload. Input tokens are the prompt (system, history, tools, retrieved chunks). Output tokens are the completion. Eval reports often add tokens per successful task and estimated cost $= n_{\text{in}}p_{\text{in}} + n_{\text{out}}p_{\text{out}}$ (+ cache/batch/reasoning tiers). Counts are tokenizer-specific (Vol. 11 tokens, Vol. 12 encodings).

Input vs Output: Why the Split Matters

Bucket	What it is	Typical price	Eval implication
Input	System + user + tools + RAG + history	Lower \$ / 1M	Prompt bloat, retrieval dump size
Output	Generated tokens (and sometimes reasoning)	Higher \$ / 1M	Verbosity, CoT length, E2E latency
Cached input	Prefix / prompt-cache hits	Discounted input	Must label cache-on vs cache-off runs
Batch	Offline jobs	Discounted both	Not comparable to interactive QPS

Output is expensive because decode is sequential GPU time (see latency). A RAG prompt of 8K input tokens with a 40-token answer can still be cheaper than a 400-token rambling completion. Optimizing the wrong side wastes the Vol. 13 playbook.

Usage as an Evaluation Metric

Ops view (Vol. 13 / 18)

Budgets, quotas, alerts
Per-tenant dashboards
Rate limits and 429s

Eval view (Vol. 19)

Mean in/out tokens per item
Tokens per correct answer
Cost to hit a quality bar

Fair compare

Same task + success criterion
Same max_tokens policy
Disclose cache / batch / SKU

Why track tokens in eval

Catches “smarter” models that just write novels.
Makes RAG chunk budgets visible.
Links quality work to unit economics.

Pitfalls

Word count $\neq$ token count.
Hidden reasoning tokens under-counted if you only log visible text.
Truncation looks cheap and fails the task.

From Usage Object to Cost

# pip install tiktoken
# Local estimate (Vol. 12/13) vs billed usage from the API.

import tiktoken

# Example list prices — replace with your vendor card; do not hardcode forever.
PRICE_PER_M = {"in": 0.50, "out": 2.00}  # USD / 1M tokens (illustrative)

def estimate_cost(n_in, n_out, price=PRICE_PER_M):
    return (n_in * price["in"] + n_out * price["out"]) / 1_000_000

enc = tiktoken.get_encoding("o200k_base")  # match the model family you call

def count_prompt(messages):
    # Toy: join contents. Real chat templates add extra tokens (Vol. 13 counting).
    text = "\n".join(m["content"] for m in messages)
    return len(enc.encode(text))

messages = [
    {"role": "system", "content": "Answer in one sentence."},
    {"role": "user", "content": "What is token usage in LLM eval?"},
]
n_in_est = count_prompt(messages)

# After the call, prefer vendor usage (source of truth for billing):
# u = resp.usage
# n_in, n_out = u.prompt_tokens, u.completion_tokens
n_in, n_out = n_in_est, 48  # placeholder completion length
print({"in": n_in, "out": n_out, "usd": round(estimate_cost(n_in, n_out), 6)})

# Eval rollup: cost per successful item = sum(usd) / n_correct
# Do not compare token totals across different tokenizers.

Related Lectures

Lecture	Role
Token counting / tiktoken	How to measure locally
Input vs output tokens	Two billable buckets
I/O pricing tiers / cost estimation	$ math
Prompt caching	Discounted input; label in evals
Latency / Hallucination tests	Time $\leftrightarrow$ tokens; quality next

Common Misconception

“Token usage is only an ops/billing concern.” In Vol. 19 it is a first-class eval axis: quality per dollar and per token. Second: treating input and output as the same price. Third: comparing character counts across models. Fourth: ignoring cached vs uncached input when claiming a 40% cost win. Fifth: celebrating fewer tokens after you lowered max_tokens so hard the model never finishes the answer.

Knowledge Check

Short Answer: What are input vs output tokens? Answer: Prompt/context tokens vs generated completion tokens.
True/False: Output tokens are often priced higher per token than input. Answer: True.
Multiple Choice: Cost is roughly: (a) $n_{\text{in}}p_{\text{in}}+n_{\text{out}}p_{\text{out}}$, (b) BLEU, (c) p95 only. Answer: (a).
Short Answer: Why is token usage an eval metric, not only billing? Answer: It measures efficiency / cost-to-quality for the same task.
True/False: Word count equals token count. Answer: False—tokenizer-dependent.
Multiple Choice: Vol. 13 companion topic: (a) token management & pricing, (b) UMAP, (c) DreamBooth. Answer: (a).
Short Answer: Name one extra usage bucket besides plain in/out. Answer: Cached input, batch, or reasoning/hidden tokens.
True/False: Truncating max_tokens always improves an eval if tokens drop. Answer: False—task success may collapse.
Multiple Choice: Next lecture: (a) Hallucination tests, (b) PCA, (c) Celery. Answer: (a).
Short Answer: Prefer vendor usage or local tiktoken for billing truth? Answer: Vendor usage object (tiktoken is an estimate).

Key Takeaways

Split input vs output tokens; price them separately (Vol. 13).
Report tokens and \$ per item and per successful item.
Tokenizer, cache, batch, and hidden reasoning confound naive totals.
Efficiency evals sit beside latency; they do not replace quality tests.
Next: Hallucination tests.

Trainer’s Guide

Lab: Run the same 20 questions with a terse vs verbose system prompt. Log in/out tokens, estimate USD, and accuracy. Compute cost per correct answer. Add a RAG dump and watch input tokens explode.

Whiteboard: Two bars: input \$ vs output \$. Arrow from RAG chunk count to input; arrow from “think step by step” to output. Next arrow: hallucination tests when cheaper answers invent facts.

Recap: Token usage evals meter input vs output (and \$), linking Vol. 13 cost control to Vol. 19 quality-per-dollar. Check whether those cheaper tokens are true next with Hallucination tests.