TF-IDF

How This Lesson Fits the Module & Volume

One-hot encoding marks which words appear, but treats every vocabulary item as equally informative. In real documents, words like “the” and “and” dominate counts while rare topic words carry the signal—especially after Module 9.1 stop-word awareness.

TF-IDF (term frequency–inverse document frequency) reweights sparse document vectors so frequent-but-ubiquitous terms shrink and distinctive terms grow. It sits between raw discrete encodings and dense neural embeddings, and pairs naturally with the bag-of-words view you will formalize next.

Learning Objectives

By the end of this lesson, students should be able to:

Define term frequency (TF) and inverse document frequency (IDF).
Compute TF-IDF weights by hand for a tiny corpus.
Explain why IDF downweights words that appear in almost every document.
Build a TF-IDF feature matrix with sklearn.feature_extraction.text.TfidfVectorizer.
Compare TF-IDF sparse vectors to one-hot and to later dense embeddings.
Choose sensible preprocessing (tokenization, n-grams, max_df/min_df) for a classification task.

Definition

TF-IDF assigns each term t in document d a weight proportional to how often t occurs in d (TF) and inversely proportional to how many documents contain t (IDF). A common form is tfidf(t, d) = tf(t, d) × log((N + 1) / (df(t) + 1)) + 1 (sklearn’s smoothed variant), where N is the corpus size and df(t) is document frequency.

Intuition: Frequency vs. Distinctiveness

Raw counts say “this word appeared a lot here.” IDF adds “and it is rare across the corpus.” A word that appears 10 times in one sports article but almost nowhere else gets a high weight; a word that appears in every article gets a low weight even if it is frequent locally.

Component	Question it answers	Effect
TF	How often is t in this document?	Boosts locally frequent terms
IDF	How rare is t across the corpus?	Penalizes ubiquitous terms
TF × IDF	How distinctive is t for this document?	Balanced importance weight

Worked Mini-Corpus

Documents: D1 = “cat sat mat”, D2 = “dog sat mat”, D3 = “cat dog”. Term “sat” appears in 2 of 3 documents; “cat” in 2 of 3; each document has short equal TFs. After IDF, shared words shrink relative to words that better separate topics—exactly the pattern linear classifiers exploit.

1. Tokenize

Reuse Module 9.1 tokenization ideas.

2. Count

Build term–document frequencies.

3. Weight

Multiply by IDF (often L2-normalize rows).

4. Model

Feed sparse vectors to logistic regression, SVM, etc.

Code: sklearn TfidfVectorizer

from sklearn.feature_extraction.text import TfidfVectorizer

docs = [
    "the cat sat on the mat",
    "the dog sat on the log",
    "cats and dogs are pets",
]
vec = TfidfVectorizer(stop_words="english", ngram_range=(1, 2), min_df=1)
X = vec.fit_transform(docs)          # sparse CSR matrix
print(X.shape)                       # (3, n_features)
print(vec.get_feature_names_out()[:8])
print(X[0].toarray().round(3))       # TF-IDF row for doc 0

TF-IDF vs. One-Hot vs. Dense Embeddings

One-hot / binary BoW

Presence only (0/1).
No frequency or rarity.
Fast baseline.

Weighted sparse counts.
Corpus-aware importance.
Still no word similarity.

Word2Vec / GloVe

Dense, fixed-size vectors.
Semantic geometry.
Needs more data/compute.

Practical Knobs

Useful settings

min_df / max_df to drop rare noise and near-universal terms.
ngram_range=(1,2) for short phrases.
sublinear_tf=True (log TF) for long documents.

Limitations

Ignores word order beyond n-grams.
Synonyms remain unrelated dimensions.
Vocabulary can still be huge and sparse.

Common Misconception

“TF-IDF is a neural embedding.” TF-IDF produces high-dimensional sparse document (or term) vectors with hand-crafted weights. It does not learn a low-dimensional semantic space. Dense methods like Word2Vec and trainable embedding layers come later.

Knowledge Check

Short Answer: What do TF and IDF stand for? Answer: Term Frequency and Inverse Document Frequency.
True/False: IDF is high when a term appears in almost every document. Answer: False—IDF is low for ubiquitous terms.
Multiple Choice: TF-IDF mainly improves: (a) word-order modeling, (b) term importance weighting, (c) GPU speed. Answer: (b).
Short Answer: Why might “the” get a low TF-IDF weight? Answer: High document frequency → low IDF.
True/False: sklearn’s TfidfVectorizer returns a dense NumPy array by default. Answer: False—it returns a sparse matrix.
Multiple Choice: Compared with one-hot presence vectors, TF-IDF: (a) uses continuous weights, (b) always uses 300 dimensions, (c) requires gensim. Answer: (a).
Short Answer: Name one hyperparameter that limits rare terms. Answer: min_df (or max_features).
Short Answer: Does TF-IDF capture that “car” and “automobile” are similar? Answer: No—they remain separate dimensions.
Multiple Choice: A typical next model after TF-IDF features is: (a) logistic regression / SVM, (b) only transformers, (c) k-means on pixels. Answer: (a).
True/False: TF-IDF still discards most word-order information. Answer: True (unless using n-grams carefully).

Key Takeaways

TF-IDF weights terms by local frequency and global rarity.
It is a strong classical baseline for document classification and retrieval.
Vectors remain sparse and non-semantic across synonyms.
sklearn’s TfidfVectorizer is the standard production tool in this curriculum.
Next, formalize the count model behind these weights: Bag of Words.

Trainer’s Guide

Hands-on idea: Fit TfidfVectorizer on three short docs; print top-weighted terms per document and discuss why stop words vanished.

Discussion prompt: When would raw counts beat TF-IDF? (Very short texts, or tasks where common words are the signal.)

Recap: TF-IDF turns term counts into distinctive document weights. Continue with Bag of Words.