← Master Index
Vol. 09 Module 9.1 Lecture

Stop Words

NLP Basics

How This Lesson Fits the Module & Volume

After tokenization (and optional sentence segmentation), classical NLP pipelines often drop high-frequency function words—stop words—before building bag-of-words or TF-IDF features in Module 9.2.

Neural models usually keep stop words because context matters. This lecture teaches when filtering helps, when it hurts, and how to customize lists for your domain.

Learning Objectives

By the end of this lesson, students should be able to:

  • Define stop words and give English examples.
  • Explain why stop-word removal reduces vocabulary noise for count-based models.
  • Filter tokens with NLTK and spaCy stop lists.
  • Decide when not to remove stop words (negation, transformers, legal text).
  • Build a domain-specific stop list (e.g., product UI chrome words).
  • Connect stop-word policy to TF-IDF and embedding workflows.
Definition

Stop words are extremely common tokens (often function words like the, is, and, of) that carry little discriminative value for some retrieval and classification tasks. A stop list is the set of such tokens removed during preprocessing.

Why Filter Them?

In bag-of-words space, the appears everywhere and adds dimensions without separating classes. Removing stop words shrinks feature vectors and can improve sparse linear models. TF-IDF already down-weights corpus-wide terms, so stop lists are optional but still common.

PipelineTypical stop-word policy
Bag-of-words / keyword searchRemove aggressive stop list
TF-IDF + logistic regressionOptional; IDF already helps
Sentiment with negationKeep not, never, no
Transformers / GRUs on raw textUsually keep all tokens
POS / parsing / NERDo not remove before tagging

Filtering in Code

import nltk import spacy from nltk.corpus import stopwords nltk.download("stopwords", quiet=True) nltk.download("punkt_tab", quiet=True) from nltk.tokenize import word_tokenize stops = set(stopwords.words("english")) # Keep negation for sentiment-style tasks keeps = {"no", "not", "nor", "never"} stops -= keeps text = "This movie was not good and the plot is thin." tokens = [t.lower() for t in word_tokenize(text) if t.isalpha()] filtered = [t for t in tokens if t not in stops] print(filtered) # keeps 'not', drops 'this', 'was', 'and', 'the', 'is' nlp = spacy.load("en_core_web_sm") print([t.text for t in nlp(text) if not t.is_stop and t.is_alpha])

Generic vs Domain Stop Lists

Generic lists

  • NLTK / spaCy English defaults.
  • Fast to apply.
  • May drop useful words.

Domain lists

  • Add “please”, “thanks” in tickets.
  • Remove product chrome words.
  • Review with precision/recall.

Statistical lists

  • Top-N DF terms as stops.
  • Fits your corpus.
  • Risk: dropping class keywords.

Benefits

  • Smaller sparse feature spaces.
  • Clearer keyword / search signals.
  • Faster classical training.

Risks

  • Destroys negation and modality.
  • Hurts phrase queries (“to be or not”).
  • Unnecessary for most neural nets.
Common Misconception

“Always remove stop words before every NLP model.” That rule dates to sparse lexical features. Modern contextual models and many linguistic pipelines need those words. Treat stop-word removal as a hyperparameter for classical features, not a universal law.

Knowledge Check

  1. Short Answer: What are stop words? Answer: Very common tokens (often function words) removed because they add little discriminative value in some tasks.
  2. True/False: Transformers usually require stop-word removal. Answer: False.
  3. Multiple Choice: For sentiment, you should often keep: (a) only nouns, (b) negation words like not, (c) HTML tags. Answer: (b).
  4. Short Answer: Why do bag-of-words models benefit from stop lists? Answer: High-frequency function words inflate dimensions without separating classes.
  5. True/False: spaCy marks stop words via token.is_stop. Answer: True.
  6. Multiple Choice: Removing stop words before POS tagging is: (a) recommended, (b) harmful to linguistic context, (c) required for BPE. Answer: (b).
  7. Short Answer: Give one domain stop-word example for support tickets. Answer: e.g., please, thanks, hello, regards (task-dependent).
  8. True/False: TF-IDF already dampens corpus-wide terms, so stop lists are optional. Answer: True.
  9. Multiple Choice: Stop lists are primarily a tool for: (a) dependency labels, (b) classical lexical features / search, (c) GPU kernels. Answer: (b).
  10. Short Answer: What morphological reduction is often applied after filtering? Answer: Stemming or lemmatization.

Key Takeaways

  • Stop words are common tokens filtered mainly for sparse lexical models.
  • Customize lists; preserve negation when polarity matters.
  • Do not strip stops before tagging, parsing, or most neural encoders.
  • Policy belongs with your Module 9.2 vectorization choices.
  • Next, Stemming reduces words to crude stems.
Trainer’s Guide

Hands-on idea: Train a quick TF-IDF classifier with and without stop-word removal; compare accuracy and top features.

Discussion prompt: Should “not” ever be a stop word? In which products?

Recap: Stop-word filtering is a classical feature choice, not a neural default. Continue with Stemming.