Vocabulary Building — Module 12.1

How This Lesson Fits the Module & Volume

You can now run BPE, WordPiece, SentencePiece, and tiktoken. Vocabulary building is the systems decision: when to reuse a public vocab, when to train a domain tokenizer, how to pick |V|, and how to evaluate fertility (tokens per word) before freezing the contract with the model.

Volume 11’s vocabulary concept becomes an engineering checklist here—feeding directly into special-token design next.

Learning Objectives

By the end of this lesson, students should be able to:

List the inputs to vocab training: corpus, algorithm, |V|, special tokens.
Choose reuse vs train-from-scratch based on domain shift and data volume.
Measure tokenization fertility and UNK/byte-fallback rates.
Train a domain BPE/Unigram vocab with Hugging Face tokenizers or SentencePiece.
Explain why extending a vocab requires resizing embedding and LM-head layers.
Plan reserved ID ranges for control tokens before training.

Definition

Vocabulary building is the offline process of constructing the finite token inventory and segmentation rules (merges / Unigram table) from a representative corpus, including reserved special tokens, then versioning those artifacts with the model they will train.

Decision Framework

Situation	Prefer	Why
General chat / match a base LM	Reuse base tokenizer	Keeps embeddings aligned
Heavy domain jargon / code dialects	Train or extend vocab	Lower fertility, better learning
Multilingual without spaces	SentencePiece Unigram/BPE	Language-agnostic training
API-only OpenAI stack	tiktoken encodings	You cannot retrain their vocab

Corpus Hygiene

Match production languages
Include code, numbers, UI strings
Deduplicate boilerplate

Size Knobs

|V| ≈ 32k–256k common
Larger → shorter seqs, bigger tables
Reserve slots for specials

Eval Metrics

Tokens / word (fertility)
UNK or long byte runs
Round-trip on held-out docs

Code: Domain BPE + Fertility Check

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import ByteLevel

tok = Tokenizer(BPE(unk_token=None))
tok.pre_tokenizer = ByteLevel(add_prefix_space=False)
trainer = BpeTrainer(
    vocab_size=8000,
    special_tokens=["", "", "", ""],
)
tok.train(files=["domain_corpus.txt"], trainer=trainer)
tok.save("domain_bpe.json")

def fertility(tokenizer, texts):
    n_tok, n_word = 0, 0
    for t in texts:
        n_tok += len(tokenizer.encode(t).ids)
        n_word += max(1, len(t.split()))
    return n_tok / n_word

held_out = ["EGFR inhibitor dosing protocol", "SELECT * FROM claims;"]
print("tokens/word:", fertility(tok, held_out))

When Building Pays Off

Domain tokens become single IDs
Shorter contexts → cheaper train/infer
Fewer brittle byte fragments

Costs & Risks

Cannot hot-swap onto old weights
Need enough clean corpus
Must resize & retrain embeddings

Common Misconception

“We can train a new tokenizer and keep the old embedding matrix as-is.” New IDs need new rows. Even “adding a few tokens” requires resizing embeddings (and usually the LM head), initializing new rows carefully, and continuing training—otherwise those IDs are random noise.

Knowledge Check

Short Answer: Name three inputs to vocabulary building. Answer: Corpus, algorithm (BPE/Unigram/…), target |V| / specials (any solid trio).
True/False: Reusing a base LM tokenizer is often correct when continuing from that LM. Answer: True.
Multiple Choice: Fertility usually means: (a) GPU FLOPs, (b) tokens per word, (c) dropout rate. Answer: (b).
Short Answer: Why reserve special-token IDs early? Answer: So control tokens have stable IDs and are not overwritten by merges.
True/False: Extending vocab never requires changing model parameters. Answer: False.
Multiple Choice: API-only GPT stacks force you to: (a) retrain tiktoken, (b) live with published encodings, (c) use WordPiece only. Answer: (b).
Short Answer: What corpus mistake inflates useless merges? Answer: Heavy duplicated boilerplate / non-representative text.
Short Answer: What must you ship with a custom vocab? Answer: Tokenizer artifacts (vocab/merges/model) versioned with weights.
Multiple Choice: Domain medical jargon often motivates: (a) smaller irrelevant vocab only, (b) custom/extended vocab, (c) deleting BPE. Answer: (b).
True/False: Lower fertility is always better regardless of |V| cost. Answer: False—balance against embedding/softmax size.

Key Takeaways

Vocab building is a deliberate systems choice, not a default checkbox.
Evaluate fertility and coverage on held-out domain text.
Reuse base tokenizers when staying aligned with pretrained weights.
New tokens imply resized embeddings and continued training.
Next: Special Tokens—control IDs that steer models and chat formats.

Trainer’s Guide

Workshop: Teams pick reuse vs retrain for (1) legal RAG on Llama, (2) bilingual support bot; defend with fertility numbers.

Demo: Add 100 domain tokens to a tiny GPT-2 clone and show random outputs before fine-tuning.

Recap: Build vocabularies from representative data, measure fertility, and keep them locked to weights. Continue with Special Tokens.