← Master Index
Vol. 13 Module 13.3 Lecture

Token Counting

Token Management & Usage (added)

How This Lesson Fits the Module & Volume

Module 13.3 turns prompting craft into operational token hygiene. Before budgeting or pricing, you must count tokens with the same encoding the model uses—typically tiktoken for OpenAI-class APIs (Vol. 12).

This lecture is the measurement foundation: wrong counts cascade into overflow, silent truncation, and cost surprises. It builds on Vol. 11 context window ideas and Vol. 12 tokenization.

Learning Objectives

By the end of this lesson, students should be able to:

  • Explain why token count ≠ word or character count.
  • Count tokens with tiktoken using encoding_for_model.
  • Account for chat/role framing overhead beyond raw string length.
  • Log prompt_tokens from API usage objects and reconcile with local counts.
  • Choose counting strategy for system, tools, history, and user turns.
  • Detect when a wrong encoding invalidates budgets.
Definition

Token counting is measuring how many tokens a string (or message list) occupies under a specific tokenizer/encoding. Providers bill and enforce limits in tokens, not words.

Why Count Before You Call

Every completion request occupies part of the model’s context budget and contributes to cost. Counting locally lets you reject oversized prompts, resize RAG chunks, and estimate spend before the network round-trip.

MetricUseful for?Risk if used alone
Characters / wordsRough UX estimatesDiverges wildly by language & encoding
Local tokenizer countPre-flight budgetsMisses chat framing / tool schema tokens
API usage fieldsGround truth after callToo late for prevention

Code: Count with tiktoken

import tiktoken enc = tiktoken.encoding_for_model("gpt-4o") def count_text(s: str) -> int: return len(enc.encode(s)) system = "You are a concise assistant." user = "Summarize the quarterly report in 5 bullets." print("system:", count_text(system)) print("user:", count_text(user)) print("naive sum:", count_text(system) + count_text(user)) # Real chat APIs add per-message framing tokens; treat local sum as a lower bound.

Local count

  • Fast pre-check
  • Needs correct encoding
  • May undercount chat wrappers

Provider usage

  • Authoritative billable
  • Returned after success
  • Use for reconciliation

Shared rule

  • Pin encoding by model
  • Version the counter
  • Never mix GPT-2 with GPT-4o

Strengths

  • Prevents avoidable 400/overflow errors
  • Enables cost forecasts
  • Stabilizes RAG chunk sizes

Tradeoffs

  • Chat overhead easy to miss
  • Encoding drift across model gens
  • Multimodal tokens need separate rules
Common Misconception

“English words ≈ tokens, so I can skip tiktoken.” Subword BPE splits punctuation, code, and non-English text unpredictably. Always count with the model’s encoding.

Knowledge Check

  1. Short Answer: What unit do LLM APIs primarily meter? Answer: Tokens (under a model-specific encoding).
  2. True/False: Character length is a reliable substitute for token count. Answer: False.
  3. Multiple Choice: Best pre-flight counter for OpenAI models: (a) len(text.split()), (b) tiktoken, (c) PDF page count. Answer: (b).
  4. Short Answer: Why might API prompt_tokens exceed a naive string encode sum? Answer: Chat role/framing (and tools) add overhead tokens.
  5. True/False: encoding_for_model picks an encoding matched to the model name. Answer: True.
  6. Multiple Choice: Wrong encoding mainly causes: (a) prettier prose, (b) wrong length/cost/limits, (c) free tokens. Answer: (b).
  7. Short Answer: Name one Vol. 12 tool for OpenAI token accounting. Answer: tiktoken.
  8. Short Answer: How does counting relate to the Vol. 11 context window? Answer: Counts must fit (and reserve space) within the window.
  9. Multiple Choice: When should you reconcile local vs API counts? (a) Never, (b) After sample calls in staging, (c) Only on weekends. Answer: (b).
  10. True/False: Token counting is optional once pricing is known. Answer: False—limits and quality still depend on it.

Key Takeaways

  • Tokens—not words—drive limits and bills.
  • Use tiktoken (or the provider’s counter) matched to the model.
  • Treat chat framing as extra tokens.
  • Reconcile local estimates with usage objects.
  • Next: Token Budgeting—allocating the counted space.
Trainer’s Guide

Lab: Encode the same multilingual FAQ with tiktoken vs a GPT-2 HF tokenizer; report % difference.

Discussion: Who owns the encoding pin in a multi-model gateway?

Recap: Token counting is the meter for every later 13.3/13.4 decision. Continue with Token Budgeting.