Tokenizer — Module 11.1

How This Lesson Fits the Module & Volume

You know tokens and the fixed vocabulary. The tokenizer is the reversible (ideally) map between raw text and token ID sequences. Vol. 09’s tokenization lecture surveyed classical and subword methods; here we treat the tokenizer as a production artifact shipped with every Hugging Face checkpoint.

Wrong tokenizer → wrong IDs → garbage embeddings and broken generation. It also determines how many tokens land in the context window.

Learning Objectives

By the end of this lesson, students should be able to:

Describe encode (text → IDs) and decode (IDs → text) as the tokenizer’s primary API.
Compare BPE, WordPiece, and Unigram at a practical level.
Use AutoTokenizer for padding, truncation, batching, and special tokens.
Explain why each pretrained model must use its matched tokenizer.
Recognize whitespace markers (e.g. Ġ) and normalization side effects.
Apply tokenizer settings that affect training batches and chat templates.

Definition

A tokenizer is the component that converts strings into sequences of vocabulary IDs (and back), using a learned or rule-based segmentation scheme plus a fixed vocabulary file. In LLM stacks it is versioned alongside model weights.

Algorithm Families

Algorithm	Idea	Seen in
Byte-Pair Encoding (BPE)	Iteratively merge frequent adjacent pairs	GPT-2/3, many LLMs
Byte-level BPE	BPE over bytes → full Unicode coverage	GPT-2, RoBERTa, Llama-ish stacks
WordPiece	Likelihood-driven merges; ## continuation marks	BERT, DistilBERT
Unigram LM	Prune a large seed vocab by loss	SentencePiece (T5, many MT)

Normalize

Unicode, lowercasing (optional).

Pre-tokenize

Split on rules / spaces.

Model

BPE / WordPiece / Unigram.

Post-process

Special tokens, padding.

Code: AutoTokenizer Batch Encode

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")
# GPT-2 has no pad by default; common training fix:
tok.pad_token = tok.eos_token

batch = [
    "Tokenizer encode/decode.",
    "Short.",
]
enc = tok(
    batch,
    padding=True,
    truncation=True,
    max_length=32,
    return_tensors="pt",
)
print(enc["input_ids"])
print(enc["attention_mask"])

for row in enc["input_ids"]:
    print(tok.convert_ids_to_tokens(row))
    print(tok.decode(row, skip_special_tokens=True))

Matched Pairs and Chat Templates

Always Match

Same hub revision as weights.
Do not mix BERT tok + GPT model.
Resize embeddings if you extend vocab.

Training Details

padding_side: left for some causal gens.
attention_mask ignores PAD positions.
Labels use ignore_index on pads.

Chat Models

apply_chat_template formats roles.
Wrong template → weak instruction following.
Special control tokens must exist.

Strengths of Modern Tokenizers

Open-vocabulary coverage.
Fast Rust implementations (HF tokenizers).
Serializable vocab + merge rules.

Tradeoffs

Opaque segmentations for humans.
Domain mismatch (code, medicine) inflates length.
Normalization can alter meaning (case, NFKC).

Common Misconception

“decode(encode(text)) always returns the exact original string.” Round-trips can change whitespace, Unicode normalization, or special-token insertion. Treat encode/decode as lossy with respect to surface form even when information for LM training is preserved. Never assume bitwise string identity after a round-trip.

Knowledge Check

Short Answer: What are the two primary tokenizer operations? Answer: Encode (text → IDs) and decode (IDs → text).
True/False: Any tokenizer can be paired with any Transformer checkpoint safely. Answer: False.
Multiple Choice: BPE builds a vocab by: (a) random splits, (b) merging frequent pairs, (c) POS tags. Answer: (b).
Short Answer: Why set pad_token for GPT-2 in batched training? Answer: GPT-2 ships without a pad token; padding needs a defined ID (often EOS reused).
True/False: attention_mask marks which positions are real tokens vs padding. Answer: True.
Multiple Choice: WordPiece continuation pieces often start with: (a) Ġ, (b) ##, (c) </w>. Answer: (b).
Short Answer: What does byte-level BPE buy you? Answer: Ability to represent any Unicode string via bytes without a classic UNK hole.
Short Answer: Why do chat templates matter? Answer: They insert the exact role/special-token format the model was instruction-tuned on.
Multiple Choice: Truncation is needed when: (a) |V| is large, (b) tokenized length exceeds max_length / context, (c) dropout is on. Answer: (b).
True/False: Tokenizer round-trips always preserve exact whitespace and Unicode. Answer: False.

Key Takeaways

The tokenizer maps text ↔ token IDs using a fixed vocabulary and algorithm (BPE/WordPiece/Unigram).
Always load the tokenizer that matches the model checkpoint.
Padding, truncation, masks, and chat templates are part of real systems—not afterthoughts.
Round-trips can be surface-lossy; validate with the same tokenizer you ship.
Next: Embedding—turning each ID into a learnable vector.

Trainer’s Guide

Hands-on idea: Break a batch encode deliberately (swap in bert-base-uncased tokenizer with GPT-2 model) and show the collapse in decoded “predictions.”

Discussion prompt: When would you train a new tokenizer for a domain instead of reusing GPT-2’s?

Recap: Tokenizers are the text–ID interface of an LM; match them to weights and handle batching carefully. Continue with Embedding.