WordPiece — Module 12.1

How This Lesson Fits the Module & Volume

After classic BPE, WordPiece is the algorithm you meet whenever you open a BERT, DistilBERT, or Electra checkpoint. It looks similar to BPE (subwords, merges) but chooses pieces with a likelihood-inspired criterion and marks continuation pieces with ##.

Understanding WordPiece prevents silent bugs when mixing BERT-family tokenizers with GPT-style models and clarifies why BERT tokenization is case- and accent-sensitive depending on the checkpoint.

Learning Objectives

By the end of this lesson, students should be able to:

Contrast WordPiece’s merge criterion with frequency-only BPE.
Interpret ## continuation markers when inspecting tokens.
Encode and decode with AutoTokenizer for a BERT checkpoint.
Explain WordPiece’s greedy longest-match encoding behavior.
Identify when UNK still appears despite subword vocabularies.
State why BERT tokenizers must not be swapped onto decoder-only LMs.

Definition

WordPiece is a subword tokenization algorithm (originally from Google ASR/NMT, popularized by BERT) that builds a vocabulary by iteratively adding merges that most improve a likelihood objective over the training corpus. Continuation subwords are typically prefixed with ##, and encoding prefers the longest matching vocabulary piece.

BPE vs WordPiece

Aspect	BPE	WordPiece
Merge score	Pair frequency	Likelihood / LM-style gain
Continuation mark	Often Ġ / </w> schemes	`##` prefix
Famous in	GPT-2 lineage	BERT family
UNK behavior	Rare with byte-level	Explicit `[UNK]` possible

Surface Clues

playing → play, ##ing
[CLS] / [SEP] framing
Optional lowercasing

Encoding Rule

Greedy longest-match from left
Fall back to shorter pieces
Else emit [UNK]

Training Stack

BasicTokenizer + WordPiece
Punctuation splits first
Vocab file: vocab.txt

Code: Inspect BERT WordPiece

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
text = "Tokenization with WordPiece is fun."
enc = tok(text)
print(enc["input_ids"])
print(tok.convert_ids_to_tokens(enc["input_ids"]))
print(tok.decode(enc["input_ids"]))

# Continuation marker demo
print(tok.tokenize("unbelievable"))
# Often something like: ['un', '##believ', '##able']

Strengths

Stable BERT ecosystem tooling
Readable ## continuations
Works well with MLM pretraining

Tradeoffs

Not byte-complete; UNK remains
Case variants need cased models
Different contract than GPT BPE

Common Misconception

“WordPiece is just BPE with ## painted on.” The marker is only surface syntax. Training uses a different merge objective, and encoding is longest-match against a fixed vocab file—not GPT-2’s merge-list application. Treating them as interchangeable corrupts IDs.

Knowledge Check

Short Answer: What prefix marks a WordPiece continuation? Answer: ##.
True/False: WordPiece merge selection is pure pair frequency like classic BPE. Answer: False—it uses a likelihood-inspired criterion.
Multiple Choice: WordPiece is iconic in: (a) BERT, (b) CLIP only, (c) k-means. Answer: (a).
Short Answer: Name BERT’s sequence boundary special tokens. Answer: [CLS] and [SEP] (also [PAD], [MASK], [UNK]).
True/False: bert-base-uncased preserves original casing. Answer: False.
Multiple Choice: Encoding prefers: (a) shortest piece, (b) longest vocab match, (c) random piece. Answer: (b).
Short Answer: When does WordPiece emit [UNK]? Answer: When no vocabulary piece can cover a span (after basic splits).
Short Answer: Why not use a BERT tokenizer with GPT-2 weights? Answer: Different vocab/IDs and special-token contracts—embeddings misalign.
Multiple Choice: Typical WordPiece vocab file name: (a) merges.txt, (b) vocab.txt, (c) tiktoken.model. Answer: (b).
True/False: ##ing means the piece starts a new word. Answer: False—it continues the previous word.

Key Takeaways

WordPiece builds subwords with a likelihood-flavored objective and ## continuations.
Encoding is greedy longest-match; UNK is still possible.
BERT-family models require their matched WordPiece tokenizer.
Do not confuse WordPiece markers with GPT-2’s Ġ whitespace encoding.
Next: SentencePiece—language-agnostic training without whitespace dependence.

Trainer’s Guide

Demo: Tokenize the same sentence with bert-base-uncased and gpt2; compare token strings side by side.

Prompt: When is a cased WordPiece model worth the vocab cost?

Recap: WordPiece is BERT’s subword engine—likelihood merges, ## pieces, longest match. Continue with SentencePiece.