Tokenization — Module 9.1

How This Lesson Fits the Module & Volume

You now know what a token is. Tokenization is the process that turns cleaned corpus text into a sequence of those tokens. Every classical feature (bag-of-words, TF-IDF) and every neural encoder (GRU from Vol. 08, transformers later) depends on this step.

This lecture compares whitespace, rule-based (spaCy/NLTK), and subword (BPE / WordPiece / Unigram) tokenizers used in production AI stacks, and prepares you for sentence segmentation and Module 9.2 vectorization.

Learning Objectives

By the end of this lesson, students should be able to:

Explain tokenization as the mapping from raw/cleaned text to token sequences.
Compare whitespace, linguistic, and subword tokenization strategies.
Tokenize text with NLTK, spaCy, and Hugging Face tokenizers / AutoTokenizer.
Describe BPE/WordPiece at a practical level (merge frequent pairs; handle rare words).
Choose a tokenizer that matches the pretrained model or classical pipeline.
Measure impacts on vocabulary size, OOV, and sequence length.

Definition

Tokenization is the process of segmenting text into a sequence of tokens according to a defined scheme (rules, statistical merges, or a learned vocabulary). The same string can yield different token sequences under different tokenizers.

Strategy Comparison

Strategy	How it works	Strength	Weakness
Whitespace	Split on spaces	Fast, simple	Breaks on punctuation / CJK
Rule / linguistic	spaCy, NLTK `word_tokenize`	Good for POS/NER pipelines	Language-specific; large vocab
Subword (BPE etc.)	Learn merge rules on corpus	Rare words → pieces; fixed vocab	Less linguistically transparent
Character	One char per token	Almost no OOV	Very long sequences

Classical Tokenizers

import nltk
import spacy
from nltk.tokenize import word_tokenize

nltk.download("punkt_tab", quiet=True)
text = "Don't stop—GPU #42 costs $1,999.50 in NYC."

print(word_tokenize(text))
# ["Do", "n't", "stop", "—", "GPU", "#", "42", "costs", "$", "1,999.50", "in", "NYC", "."]

nlp = spacy.load("en_core_web_sm")
print([t.text for t in nlp(text)])

Subword Tokenization for Neural Models

Byte-Pair Encoding (BPE) starts from characters (or bytes) and repeatedly merges the most frequent adjacent pairs until a target vocabulary size is reached. WordPiece uses a similar subword idea but selects merges by likelihood and encodes with longest-match greedy decoding (see Vol. 12 WordPiece). Rare words decompose into reusable pieces, which is why transformers rarely need an enormous word vocabulary.

from transformers import AutoTokenizer

bert = AutoTokenizer.from_pretrained("bert-base-uncased")
gpt = AutoTokenizer.from_pretrained("gpt2")

s = "Tokenization is essential for NLP."
print(bert.tokenize(s))
print(gpt.tokenize(s))
print(bert(s)["input_ids"])  # includes special tokens for BERT

Matching Tokenizer to Model

Classical ML

spaCy / NLTK word tokens.
Then counts / TF-IDF (9.2).
Often lowercase + stop-word filter.

Pretrained Transformer

Must use that model’s tokenizer.
Wrong tokenizer = garbage IDs.
Respect max length & special tokens.

Custom RNN/GRU

Build vocab from train split only.
Reserve UNK / PAD / EOS.
Subword often still wins on OOV.

Subword Pros

Fixed vocab with low OOV.
Shares pieces across rare words.
Standard for modern LLMs.

Subword Cons

Harder to explain to linguists.
Can split meaningful affixes oddly.
Must stay paired with its model.

Common Misconception

“I can tokenize with spaCy, then feed those strings into BERT.” BERT expects its own WordPiece IDs, including special tokens and casing rules. Re-tokenizing with a mismatched scheme silently destroys pretrained alignment.

Knowledge Check

Short Answer: What does tokenization produce? Answer: A sequence of tokens (and usually integer IDs) from text.
True/False: Whitespace splitting is adequate for all languages. Answer: False—many languages lack space-separated words.
Multiple Choice: BPE primarily helps by: (a) deleting stop words, (b) building a merge vocabulary that splits rare words, (c) parsing dependencies. Answer: (b).
Short Answer: Why must you use a pretrained model’s own tokenizer? Answer: Embedding rows align to that tokenizer’s vocabulary and special tokens.
True/False: NLTK word_tokenize and GPT-2 BPE will always agree. Answer: False.
Multiple Choice: Building vocab from the test set causes: (a) better fairness, (b) leakage / optimistic metrics, (c) slower GPUs. Answer: (b).
Short Answer: Name one benefit and one cost of character tokenization. Answer: Benefit: almost no OOV; cost: very long sequences.
True/False: Special tokens like [CLS] are part of BERT tokenization output. Answer: True.
Multiple Choice: For bag-of-words, engineers typically use: (a) GPT byte tokens only, (b) word-level tokens, (c) parse trees. Answer: (b).
Short Answer: What lecture follows for finding sentence boundaries? Answer: Sentence segmentation.

Key Takeaways

Tokenization maps text to token sequences; the scheme is a first-class design choice.
Linguistic tokenizers suit classical pipelines; subword tokenizers dominate neural NLP.
Always pair a pretrained model with its exact tokenizer.
Vocab must be fit on training data only.
Next, Sentence Segmentation finds sentence boundaries inside documents.

Trainer’s Guide

Hands-on idea: Compare token counts for one paragraph under whitespace, spaCy, and bert-base-uncased; discuss billing/latency implications.

Discussion prompt: When would you train a custom BPE on domain text instead of reusing GPT-2’s tokenizer?

Recap: Tokenization is the gateway from strings to model-ready sequences. Continue with Sentence Segmentation.