← Master Index
Vol. 12 Module 12.1 Lecture

Special Tokens

Tokenization Deep Dive

How This Lesson Fits the Module & Volume

A vocabulary is not only subwords. Special tokens are reserved IDs for padding, sequence boundaries, masking, chat roles, tools, and safety rails. After vocabulary building, this lecture closes Module 12.1 by showing how those control symbols interact with attention masks, loss masks, and chat templates.

Module 12.2 then shifts from discrete IDs to continuous dense embeddings for retrieval—still depending on a correct tokenizer and special-token policy upstream.

Learning Objectives

By the end of this lesson, students should be able to:

  • Catalog common special tokens (PAD, BOS/EOS, UNK, MASK, role markers).
  • Explain how PAD interacts with attention masks and label ignore indices.
  • Add special tokens with tokenizer.add_special_tokens and resize embeddings.
  • Apply chat templates that insert model-specific control tokens.
  • Avoid treating user text that looks like specials as trusted control signals.
  • Design a minimal special-token set for a fine-tuned assistant.
Definition

Special tokens are vocabulary entries reserved for structural or control purposes rather than ordinary language content. They are referenced by name in tokenizer configs, often skipped or substituted during decode, and must remain consistent between training and inference.

Roles at a Glance

TokenRoleTypical systems
PADBatch length alignmentMost trainers
BOS / EOS / SEPBoundaries / end generationGPT, BERT, T5 variants
UNKFallback unknown pieceWordPiece, some SP setups
MASKMLM corruption markerBERT family
Role / tool tagsChat & tool protocolsInstruct / agent models

Training

  • Ignore PAD in loss
  • Mask PAD in attention
  • Learn EOS as stop signal

Serving

  • Stop on EOS / stop strings
  • Chat template fidelity
  • Strip specials on display

Security

  • User can type fake tags
  • Sanitize / escape inputs
  • Never trust client-side roles

Code: Add Specials & Resize

from transformers import AutoTokenizer, AutoModelForCausalLM name = "gpt2" tok = AutoTokenizer.from_pretrained(name) model = AutoModelForCausalLM.from_pretrained(name) # GPT-2 has no pad by default if tok.pad_token is None: tok.pad_token = tok.eos_token new_tokens = ["<|tool_call|>", "<|tool_resp|>"] n_added = tok.add_special_tokens({"additional_special_tokens": new_tokens}) model.resize_token_embeddings(len(tok)) print("added:", n_added, "pad_id:", tok.pad_token_id) # Chat-style models: # tok.apply_chat_template(messages, tokenize=True, add_generation_prompt=True)

Well-Designed Specials

  • Clear stop / role structure
  • Stable IDs across releases
  • Documented in model card

Footguns

  • Add tokens, forget resize
  • Wrong padding side for causal LM
  • Leaking specials into user UI
Common Misconception

“If the user types <|assistant|>, the model will safely switch roles.” Special tokens are just IDs. Without server-side templating and privilege boundaries, user-supplied lookalikes can confuse prompts. Always build chat formatting on the server with the official template.

Knowledge Check

  1. Short Answer: Name three special-token roles. Answer: Any three of PAD, BOS/EOS, UNK, MASK, role/tool markers.
  2. True/False: PAD positions should contribute to the next-token loss. Answer: False—usually ignored via ignore_index / masks.
  3. Multiple Choice: After add_special_tokens you must: (a) delete EOS, (b) resize embeddings, (c) disable attention. Answer: (b).
  4. Short Answer: Why does GPT-2 often reuse EOS as PAD? Answer: It ships without a dedicated pad token; reusing EOS is a common training fix.
  5. True/False: apply_chat_template inserts the model’s expected control tokens/roles. Answer: True.
  6. Multiple Choice: MASK specials are central to: (a) MLM BERT training, (b) k-means, (c) JPEG. Answer: (a).
  7. Short Answer: What is a prompt-injection risk with specials? Answer: Users forging role/tool tags that the system treats as trusted structure.
  8. Short Answer: What does left-padding help with in causal batched generation? Answer: Aligning sequences so generation continues from the rightmost real tokens.
  9. Multiple Choice: Skipping specials on decode is for: (a) prettier user-facing text, (b) changing vocab size, (c) training Adam. Answer: (a).
  10. True/False: Special-token IDs can differ across tokenizer versions even if names match. Answer: True—always pin revisions.

Key Takeaways

  • Special tokens structure padding, boundaries, masking, and chat/tool protocols.
  • Adding them requires tokenizer updates plus embedding (and often LM-head) resize.
  • Attention/loss masks must respect PAD; templates must match instruction tuning.
  • Never trust user-typed lookalike control tags.
  • Next module: Dense Embeddings—vectors for retrieval and ranking.
Trainer’s Guide

Exercise: Break a chat model by applying the wrong template; then fix it with apply_chat_template.

Security drill: Show a forged <|system|> in user content and discuss mitigations.

Recap: Special tokens are the control plane of tokenization. Continue to Module 12.2 with Dense Embeddings.