TikToken

How This Lesson Fits the Module & Volume

After byte-level BPE theory, tiktoken is the production runtime OpenAI ships for counting and encoding tokens against named encodings (cl100k_base, o200k_base, etc.). Billing, context budgeting, and RAG chunking all depend on matching the model’s encoding—not a generic GPT-2 tokenizer.

This lecture connects algorithm knowledge to API hygiene: pick the right encoding, count tokens before calls, and never assume one vocab fits every GPT generation.

Learning Objectives

By the end of this lesson, students should be able to:

Explain tiktoken as a fast BPE encoder over published encoding tables.
Map common OpenAI models to encodings (e.g. cl100k_base).
Count tokens and encode/decode with the tiktoken Python package.
Use encoding selection for cost and context-window planning.
Contrast tiktoken with Hugging Face AutoTokenizer for local checkpoints.
Recognize special-token handling differences across encodings.

Definition

tiktoken is OpenAI’s open-source, Rust-accelerated BPE library that loads named encodings—fixed vocabularies and merge rankings used by GPT-3.5/4-class and related APIs. It exposes encode, decode, and token counting without downloading full Transformer weights.

Encodings You Will Meet

Encoding	Approx \|V\|	Typical models
`r50k_base` / `p50k_base`	~50k	Older GPT-3 / Codex-era
`cl100k_base`	~100k	GPT-3.5-turbo, GPT-4 (many)
`o200k_base`	~200k	Newer GPT-4o-class

API / OpenAI encoding fidelity
Blazing count for billing
No model weights needed

HF AutoTokenizer

Matches local checkpoints
Chat templates, padding
Training-time batching

Shared Idea

Both apply BPE merges
Wrong table → wrong counts
Version your encoding name

Code: Count & Encode with tiktoken

import tiktoken

# Prefer encoding_for_model when you know the model name:
enc = tiktoken.encoding_for_model("gpt-4o")
# Or pin explicitly:
# enc = tiktoken.get_encoding("o200k_base")

text = "Token budgets decide chunk size in RAG."
ids = enc.encode(text)
print(len(ids), ids)
print(enc.decode(ids))

# Special tokens (allowed set depends on encoding / API):
# enc.encode(text, allowed_special="all")

def prompt_tokens(system: str, user: str) -> int:
    # Simplified; real chat APIs add role framing overhead
    return len(enc.encode(system)) + len(enc.encode(user))

print(prompt_tokens("You are helpful.", "Summarize BPE."))

Strengths

Accurate OpenAI token accounting
Very fast Rust core
Simple encode/decode API

Tradeoffs

Not a drop-in for every HF model
Chat overhead tokens easy to miss
Encoding names change over model gens

Common Misconception

“I can budget GPT-4 calls with the GPT-2 tokenizer.” Different encodings produce different lengths for the same string. Always use encoding_for_model (or the documented encoding) for the model you call—especially when estimating cost or fitting a context window.

Knowledge Check

Short Answer: What does tiktoken primarily provide? Answer: Fast BPE encode/decode/count for named OpenAI encodings.
True/False: cl100k_base is interchangeable with GPT-2’s vocab for counting. Answer: False.
Multiple Choice: encoding_for_model("gpt-4o") returns: (a) weights, (b) an Encoding object, (c) a BERT WordPiece. Answer: (b).
Short Answer: Why count tokens before an API call? Answer: Cost control and staying within the context window.
True/False: Chat messages may use more tokens than raw concatenation of strings. Answer: True—role/framing overhead.
Multiple Choice: tiktoken is accelerated with: (a) Prolog, (b) Rust, (c) COBOL. Answer: (b).
Short Answer: Name one encoding used by GPT-3.5/4-era models. Answer: cl100k_base (or o200k_base for newer).
Short Answer: When should you prefer HF AutoTokenizer over tiktoken? Answer: When matching a local/open checkpoint’s shipped tokenizer.
Multiple Choice: Wrong encoding mainly risks: (a) silent GPU OOM only, (b) wrong length/cost/truncation, (c) better BLEU. Answer: (b).
True/False: enc.decode(enc.encode(s)) always equals s bit-for-bit for every string. Answer: False—round-trips can still differ in edge cases; treat carefully.

Key Takeaways

tiktoken is the practical encoder for OpenAI encoding tables.
Always select encoding by model name or documented ID.
Token counts drive cost, chunking, and context packing.
Local HF models still need their own tokenizers.
Next: Vocabulary Building—designing and training a vocab from data.

Trainer’s Guide

Lab: Compare token counts for the same FAQ corpus under cl100k_base vs GPT-2 HF tokenizer; discuss RAG chunk size impact.

Discussion: Who owns encoding version pins in a multi-model gateway?

Recap: tiktoken turns byte-level BPE into production token accounting. Continue with Vocabulary Building.