Stemming chops; lemmatization looks up the canonical dictionary form—the lemma—using vocabulary knowledge and usually part-of-speech. It sits between surface tokens and higher analyses like POS tagging.
Use lemmas when you need readable normalization, better morphological grouping than stems, or features for classical models without non-word stems. Contextual embeddings later often make aggressive lemmatization optional.
Learning Objectives
By the end of this lesson, students should be able to:
- Define lemmatization and the lemma.
- Explain why POS context improves lemmatization (meeting noun vs verb).
- Lemmatize with WordNetLemmatizer and spaCy.
- Compare stemming vs lemmatization on irregular verbs and plurals.
- Choose lemmatization appropriately for search, analytics, and neural pipelines.
- Avoid lemmatizing before tasks that need surface forms (exact match, some NER).
Lemmatization maps a word form to its lemma—the canonical dictionary citation form (e.g., ran / running → run; better → good when tagged as adjective). Unlike stemming, the result is typically a valid word.
Why POS Matters
The string meeting is a noun in “the meeting ended” and a verb in “we are meeting.” A lemmatizer without POS may leave both as meeting; with POS, the verb becomes meet. That is why spaCy lemmatizes inside a full pipeline and why NLTK’s WordNet lemmatizer accepts a POS hint.
| Form | POS | Lemma |
|---|---|---|
| running | VERB | run |
| running | NOUN (the running) | running |
| mice | NOUN | mouse |
| better | ADJ | good |
| are | AUX/VERB | be |
Code: NLTK and spaCy
Choosing Normalization
Prefer lemmas
- Analytics dashboards / word clouds.
- Morphology-aware classical features.
- Need valid words for humans.
Prefer stems
- Ultra-fast search indexing.
- Acceptable non-word stems.
- Legacy IR systems.
Prefer neither
- Pretrained transformers.
- Subword BPE already shares forms.
- Exact string matching tasks.
Strengths
- Readable, linguistically motivated.
- Handles many irregular forms.
- Integrates cleanly with spaCy docs.
Tradeoffs
- Needs POS / model download.
- Slower than Porter stemming.
- Language resources required.
“Lemmatization always improves accuracy.” Collapsing Apple (org) and apple (fruit) via lowercasing + lemmatization can hurt NER and entity-sensitive tasks. Normalize only after you know which distinctions matter.
Knowledge Check
- Short Answer: What is a lemma? Answer: The canonical dictionary citation form of a word.
- True/False: Lemmatization outputs are usually valid words. Answer: True.
- Multiple Choice: Lemmatizing running as a verb typically yields: (a) runn, (b) run, (c) runningly. Answer: (b).
- Short Answer: Why pass POS to WordNetLemmatizer? Answer: The correct lemma can depend on part of speech.
- True/False: spaCy exposes lemmas on each token as
token.lemma_. Answer: True. - Multiple Choice: Compared to stemming, lemmatization is generally: (a) faster but cruder, (b) slower but more precise, (c) identical. Answer: (b).
- Short Answer: When might you skip lemmatization? Answer: Transformer/subword pipelines or tasks needing exact surface forms.
- True/False: better as an adjective lemmatizes to good. Answer: True (with proper POS).
- Multiple Choice: Lemmatization belongs closest to: (a) spectrogram filtering, (b) morphological normalization, (c) gradient clipping. Answer: (b).
- Short Answer: Which lecture supplies the POS tags lemmatizers rely on? Answer: POS Tagging.
Key Takeaways
- Lemmatization maps forms to dictionary lemmas, guided by POS.
- It is cleaner than stemming but more expensive.
- Use it for readable normalization and classical features; often skip for transformers.
- Do not blindly lemmatize entity-sensitive text.
- Next, POS Tagging labels each token’s grammatical category.
Hands-on idea: Lemmatize a paragraph with and without POS hints in NLTK; list disagreements with spaCy.
Discussion prompt: For product search, do users benefit more from stems, lemmas, or learned query expansion?
Recap: Lemmatization yields true base forms when morphology and POS are known. Continue with POS Tagging.