Sparse Embeddings — Module 12.2

How This Lesson Fits the Module & Volume

Dense embeddings shine at meaning; they stumble on exact terms. Sparse embeddings—from classic TF-IDF/BM25 bags to learned sparse retrievers (SPLADE-style)—place mass on vocabulary dimensions so lexical overlap remains first-class. Production search often keeps both.

This lesson reconnects Volume 09 classical IR with modern neural sparse vectors before hybrid fusion.

Learning Objectives

By the end of this lesson, students should be able to:

Define sparse text vectors as high-dimensional, mostly-zero representations.
Contrast BM25/TF-IDF with learned sparse neural retrievers.
Explain why inverted indexes suit sparse retrieval.
Implement a minimal TF-IDF cosine baseline in scikit-learn.
Identify query types that favor sparse over dense.
Describe expansion (e.g. SPLADE) as predicting term weights beyond surface words.

Definition

A sparse embedding represents a text as a vector in a large vocabulary-sized (or term) space where only a few coordinates are nonzero—classically term frequencies / TF-IDF / BM25 scores, and in neural systems learned term weights (sometimes with expansion terms not present in the surface string).

Classic vs Learned Sparse

Family	Dimensions	How weights arise
BoW / TF-IDF	\|vocab\|	Count × IDF heuristics
BM25	\|vocab\|	Saturated TF + length norm
Learned sparse (SPLADE…)	~tokenizer vocab	Neural logits → sparse weights

Wins

Exact keywords & IDs
Interpretable dimensions
Mature inverted-index ops

Limits

Vocabulary mismatch
Weak paraphrase recall
Huge explicit dims

Neural Sparse

Term expansion helps synonyms
Still indexable sparsely
Heavier to train/serve

Code: TF-IDF Sparse Baseline

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

docs = [
    "Order SKU-12345 ships in two days.",
    "Dense embeddings capture paraphrases.",
    "BM25 ranks by term statistics.",
]
q = "status of SKU-12345"

vec = TfidfVectorizer()
X = vec.fit_transform(docs)
q_vec = vec.transform([q])
scores = cosine_similarity(q_vec, X).ravel()
for i in scores.argsort()[::-1]:
    print(f"{scores[i]:.3f}", docs[i])
print("nnz query dims:", q_vec.nnz, "of", q_vec.shape[1])

Operational Pros

Fast Boolean / BM25 stacks
Easy debug (“why this hit?”)
Great for catalogs

Operational Cons

Synonym dictionaries needed
Language morphology pain
Misses “meaning only” queries

Common Misconception

“Sparse means outdated; dense replaced it.” Sparse lexical matching remains state-of-the-art for many enterprise queries and is a core half of hybrid retrieval. Neural sparse methods exist precisely because lexical signals still matter.

Knowledge Check

Short Answer: Why are sparse vectors called sparse? Answer: Most coordinates are zero; only a few terms fire.
True/False: BM25 is a dense sentence-transformer model. Answer: False.
Multiple Choice: Inverted indexes map: (a) terms → posting lists, (b) GPUs → batches, (c) layers → heads. Answer: (a).
Short Answer: Name one sparse failure mode. Answer: Vocabulary mismatch / paraphrase miss (or morphology).
True/False: Learned sparse retrievers can put weight on terms not in the query string. Answer: True (expansion).
Multiple Choice: SKU-12345 lookup favors: (a) sparse/lexical, (b) only CLIP, (c) random vectors. Answer: (a).
Short Answer: What does IDF down-weight? Answer: Terms that appear in many documents (common terms).
Short Answer: Name a neural sparse family mentioned in IR literature. Answer: SPLADE (or similar learned sparse models).
Multiple Choice: TF-IDF vectors live in roughly: (a) 3-D RGB, (b) vocabulary space, (c) time only. Answer: (b).
True/False: Sparse retrieval is obsolete in all RAG systems. Answer: False.

Key Takeaways

Sparse embeddings emphasize lexical term dimensions.
BM25/TF-IDF remain strong baselines; neural sparse adds expansion.
Inverted indexes make sparse search efficient and debuggable.
Use sparse when exact terms matter; combine with dense for meaning.
Next: Hybrid Embeddings.

Trainer’s Guide

Demo: Same query set on BM25 vs MiniLM; tally where each wins.

Discussion: Should product IDs be filtered into a dedicated keyword field?

Recap: Sparse vectors keep lexical precision in the retrieval toolbox. Continue with Hybrid Embeddings.