Every next-token distribution is defined over a fixed discrete set: the vocabulary. Its size |V| is the width of the LM head and of the embedding table. The context window limits sequence length; vocabulary size limits how finely text is carved into tokens.
Vol. 09 covered classical vocabularies for BoW/TF-IDF. Here the vocabulary is the shared contract between tokenizer and model—swap either side and IDs become nonsense.
Learning Objectives
By the end of this lesson, students should be able to:
- Define vocabulary as the finite set of token types a model can emit or embed.
- Explain how |V| trades off sequence length, embedding memory, and softmax cost.
- Identify special tokens (PAD, EOS, BOS, UNK, chat markers) and their roles.
- Inspect a Hugging Face tokenizer’s vocab size and decode arbitrary IDs.
- Relate subword vocabularies to OOV handling versus word-level vocabularies.
- State why model and tokenizer vocabularies must stay locked together.
A vocabulary is the ordered set of atomic symbols (token types) that an LM recognizes. Each type has an integer ID in {0,…,|V|-1}. The LM head outputs |V| logits; the embedding matrix has |V| rows.
Size Tradeoffs
| |V| regime | Pros | Cons |
|---|---|---|
| Small (char / tiny) | Almost no OOV; compact tables | Very long sequences |
| Mid (~32k subword) | Balanced length vs table size | Common LLM default |
| Large (~100k–256k) | Fewer splits; multi-lingual coverage | Bigger embedding + LM head |
| Word-level huge | Linguistically readable tokens | Severe OOV; sparse learning |
Embedding Cost
- Parameters ≈ |V| × d_model.
- Often a large fraction of small models.
- Tied input/output embeddings save space.
Softmax Cost
- Each position scores all |V| types.
- Dominates tiny models’ last layer.
- Sampled softmax rare in modern LLMs.
Special Tokens
- PAD, EOS/BOS, UNK, MASK.
- Chat: <|user|>, tool tags, etc.
- Must be in vocab to be generated.
Code: Inspect a Model Vocabulary
Vocabulary vs Corpus Types
In classical NLP, “vocabulary” sometimes means all word types in a corpus. In LMs it means the tokenizer’s fixed inventory, learned offline (BPE/WordPiece/Unigram) before LM training. Rare corpus words are expressed as sequences of subword types rather than new IDs.
Design Goals
- Cover training languages/scripts.
- Keep average tokens/word reasonable.
- Reserve IDs for control tokens.
Footguns
- Mismatched tokenizer/model checkpoints.
- Extending vocab without resizing layers.
- Forgetting special-token initialization.
“Larger vocabulary is always better.” Huge |V| shortens sequences but inflates the embedding and LM-head matrices and can starve rare IDs of gradient updates. Subword vocabularies (~30k–100k+) are a deliberate compromise—not a race to include every word form.
Knowledge Check
- Short Answer: What does |V| determine about the LM head? Answer: It has |V| output logits (one per token type).
- True/False: You can freely swap tokenizers between two LMs with different vocabularies. Answer: False—IDs must match the trained embedding/LM head.
- Multiple Choice: Embedding matrix shape is typically: (a) (d, d), (b) (|V|, d), (c) (T, |V|). Answer: (b).
- Short Answer: Name two special token roles. Answer: Any two of PAD, EOS, BOS, UNK, MASK, chat/role markers.
- True/False: Subword and byte-level vocabularies largely remove classic word-level OOV holes. Answer: True.
- Multiple Choice: A downside of very large |V| is: (a) shorter contexts always, (b) larger embedding/softmax cost, (c) no need for tokenizers. Answer: (b).
- Short Answer: What is embedding tying? Answer: Sharing (or tying) input embedding weights with the output LM-head projection.
- Short Answer: How does GPT-2 encode a rare long word? Answer: As a sequence of subword pieces from its fixed vocab.
- Multiple Choice: Classical corpus vocab vs LM vocab: (a) identical concepts, (b) LM vocab is the tokenizer’s fixed type set, (c) LM vocab changes every batch. Answer: (b).
- True/False: Adding new special tokens requires resizing embedding and LM-head parameters. Answer: True.
Key Takeaways
- The vocabulary is the finite set of token types; |V| sizes embeddings and the LM head.
- Subword inventories balance sequence length, coverage, and parameter cost.
- Special tokens are first-class vocab entries used for control and formatting.
- Tokenizer and model must share the same vocabulary mapping.
- Next: Tokens—the instances that flow through the context window.
Hands-on idea: Compare tok.vocab_size for gpt2, bert-base-uncased, and a Llama tokenizer if available; discuss multilingual pressure on |V|.
Discussion prompt: If you add a <SQL> special token for a code agent, what else must you change in the checkpoint?
Recap: Vocabulary is the LM’s finite symbol alphabet—sizing it is a core systems tradeoff. Continue with Tokens.