Token streams still contain morphological variants: running, runs, ran. For classical lexical features, collapsing variants reduces sparsity. Stemming is the fast, rule-based way to chop endings—cruder than lemmatization, but cheap and common in search and TF-IDF pipelines after stop-word filtering.
Neural embedding models (Module 9.2) often skip stemming because subword tokenizers already share pieces across forms.
Learning Objectives
By the end of this lesson, students should be able to:
- Define stemming and contrast it with lemmatization.
- Apply Porter and Snowball stemmers in NLTK.
- Recognize over-stemming and under-stemming errors.
- Explain when stemming helps sparse retrieval/classification.
- Decide when to skip stemming for neural or linguistic pipelines.
- Measure vocabulary reduction after stemming on a sample corpus.
Stemming reduces inflected or derived word forms to a crude stem by stripping affixes with heuristic rules. The stem need not be a valid dictionary word (e.g., studies → studi).
Stemming vs Lemmatization (Preview)
| Aspect | Stemming | Lemmatization |
|---|---|---|
| Method | Rule heuristics | Dictionary + POS |
| Output | May be non-word | Valid lemma |
| Speed | Very fast | Slower |
| Quality | Noisy merges | More precise |
| Typical use | Search, TF-IDF | Linguistic NLP |
Porter & Snowball in Practice
Error Modes
Over-stemming
- Distinct words → same stem.
- Example: university / universe.
- Hurts precision in search.
Under-stemming
- Related forms stay separate.
- Example: irregular verbs.
- Hurts recall.
Language limits
- English-centric rules.
- Morphology-rich languages suffer.
- Prefer language-aware tools.
When Stemming Wins
- Keyword search recall.
- Small labeled sets + bag-of-words.
- Need millisecond preprocessing.
When to Skip
- Before POS / NER / parsing.
- Transformer or subword pipelines.
- When lemmas are required for UX.
“The stem is the dictionary base form.” Stems are algorithmic chops, not lemmas. Showing studi to end users looks broken; use lemmatization when you need readable base forms.
Knowledge Check
- Short Answer: What is stemming? Answer: Heuristic affix-stripping that maps word forms to a crude stem.
- True/False: A stem must be a valid English word. Answer: False.
- Multiple Choice: Over-stemming mainly hurts: (a) GPU temperature, (b) precision by merging unrelated words, (c) UTF-8 decoding. Answer: (b).
- Short Answer: Name two NLTK stemmers. Answer: PorterStemmer and SnowballStemmer (also Lancaster).
- True/False: Stemming is usually applied before POS tagging. Answer: False—tagging needs original forms.
- Multiple Choice: Stemming vs lemmatization speed: (a) stemming faster, (b) equal, (c) lemmatization always faster. Answer: (a).
- Short Answer: Why do subword models often skip stemming? Answer: Pieces already share morphology; stemming can fight the tokenizer.
- True/False: Irregular past tense is always handled perfectly by Porter. Answer: False—under-stemming is common.
- Multiple Choice: Stemming is most associated with: (a) dependency arcs, (b) classical IR / lexical features, (c) spectrograms. Answer: (b).
- Short Answer: What is the linguistically cleaner alternative? Answer: Lemmatization.
Key Takeaways
- Stemming is fast heuristic normalization for sparse lexical pipelines.
- Expect over- and under-stemming; stems may not be real words.
- Skip stemming for linguistic analysis and most neural encoders.
- Compare vocabulary size before/after to quantify the effect.
- Next, Lemmatization maps tokens to true dictionary lemmas.
Hands-on idea: Have students find five over-stemming pairs with Porter and propose whether Snowball or lemmas fix them.
Discussion prompt: For an internal enterprise search box, is stemming worth the precision loss?
Recap: Stemming crudely collapses morphology for speed and recall. Continue with Lemmatization.