SentencePiece — Module 12.1

How This Lesson Fits the Module & Volume

BPE and WordPiece historically assumed whitespace pre-tokenization. SentencePiece treats the raw Unicode string (with an explicit space marker) as the training unit, which is essential for Japanese, Chinese, Thai, and robust multilingual models such as T5, ALBERT, and many Llama-adjacent stacks.

It also packages both BPE and Unigram LM algorithms behind one .model file—the artifact you will see next to many Hugging Face checkpoints.

Learning Objectives

By the end of this lesson, students should be able to:

Explain why SentencePiece avoids language-specific whitespace pre-tokenizers.
Distinguish SentencePiece BPE mode from Unigram LM mode.
Interpret the meta space symbol (▁) in piece strings.
Train a tiny SentencePiece model and encode/decode with the Python API.
Load SentencePiece-backed tokenizers via AutoTokenizer.
Choose Unigram vs BPE for multilingual or lossy-segmentation use cases.

Definition

SentencePiece is an unsupervised text tokenizer and detokenizer (Kudo & Richardson) that trains directly from raw sentences. It supports BPE and Unigram language model algorithms, encodes spaces as a visible piece character (often ▁), and ships a portable binary .model plus vocab.

Algorithms Inside SentencePiece

Mode	Idea	Encoding
BPE	Frequency merges like classic BPE	Greedy merges
Unigram LM	Start large; prune pieces by loss	Viterbi / sampling
char / word	Baselines for ablation	Trivial splits

Why Raw Text?

No English-centric space rules
Consistent multilingual pipelines
Reversible detokenization goal

Unigram Benefit

Probabilistic segmentations
Subword regularization (sampling)
Used by T5 / ALBERT lineages

Artifacts

spiece.model
Optional vocab export
HF wrapper still preferred in apps

Code: Train & Encode with SentencePiece

import sentencepiece as spm

# Write a tiny corpus file first, then:
spm.SentencePieceTrainer.train(
    input="corpus.txt",
    model_prefix="toy_sp",
    vocab_size=200,
    model_type="unigram",  # or "bpe"
    character_coverage=0.9995,
    user_defined_symbols=["", "", ""],
)

sp = spm.SentencePieceProcessor(model_file="toy_sp.model")
print(sp.encode("Hello world", out_type=str))
print(sp.encode("Hello world", out_type=int))
print(sp.decode(sp.encode("Hello world")))

# Meta space shows up as ▁ before words that followed whitespace
# e.g. ['▁Hello', '▁world']

Strengths

Language-agnostic training
BPE + Unigram in one toolkit
Strong detokenization story

Tradeoffs

Binary model less human-readable
Must match training normalization
Coverage params matter for CJK

Common Misconception

“▁ is just an underscore for readability.” It is a dedicated meta symbol meaning “whitespace preceded this piece” (U+2581). Stripping or replacing it casually breaks detokenization and changes token IDs.

Knowledge Check

Short Answer: Name two algorithms SentencePiece can train. Answer: BPE and Unigram LM (also char/word baselines).
True/False: SentencePiece requires a whitespace pre-tokenizer like BasicTokenizer. Answer: False.
Multiple Choice: The meta space symbol is commonly: (a) Ġ only, (b) ▁, (c) ##. Answer: (b).
Short Answer: What file extension is the trained model? Answer: Typically .model (e.g. spiece.model).
True/False: Unigram mode can sample alternate segmentations for regularization. Answer: True.
Multiple Choice: T5-style tokenizers are often: (a) WordPiece-only, (b) SentencePiece Unigram, (c) regex only. Answer: (b).
Short Answer: Why is SentencePiece popular for Japanese/Chinese? Answer: It does not rely on space-separated words.
Short Answer: What does character_coverage control? Answer: How much of the Unicode alphabet is retained vs mapped to UNK/bytes.
Multiple Choice: Detokenization aims to: (a) drop all spaces, (b) restore a surface string, (c) POS-tag. Answer: (b).
True/False: You can freely edit ▁ out of pieces without changing meaning. Answer: False.

Key Takeaways

SentencePiece trains on raw text with an explicit space marker.
BPE and Unigram modes share tooling but differ in segmentation.
Unigram enables probabilistic / sampled subword regularization.
Always ship the .model with the checkpoint.
Next: Byte-Level BPE—full Unicode coverage via bytes.

Trainer’s Guide

Lab: Train Unigram vs BPE SentencePiece on a bilingual mini-corpus; compare piece lists for the same sentence.

Discussion: When is subword regularization worth the training complexity?

Recap: SentencePiece is the language-agnostic tokenizer toolkit behind many multilingual LMs. Continue with Byte-Level BPE.