Bag of Words — Module 9.2

How This Lesson Fits the Module & Volume

You have seen one-hot word vectors and TF-IDF weights. Both assume a document can be summarized by which vocabulary items it contains. That assumption has a name: the bag-of-words (BoW) model.

BoW is the classical document representation that ignores order (a “bag”) and keeps multiset counts—or binary presence. It is the conceptual parent of TF-IDF and the baseline that distributional methods like Word2Vec leave behind when they model context windows instead of whole-document histograms.

Learning Objectives

By the end of this lesson, students should be able to:

Define the bag-of-words model and state what information it discards.
Build binary and count BoW vectors for short documents by hand.
Implement BoW with CountVectorizer and relate it to TfidfVectorizer.
Explain how n-grams partially restore local order.
Contrast document-level BoW with context-window methods (Word2Vec).
Choose when BoW is enough versus when dense embeddings are needed.

Definition

A bag-of-words representation maps a text to a vector over vocabulary dimensions where each entry is a function of how often (or whether) that word appears—typically raw counts, binary indicators, or TF-IDF weights—without encoding the sequence of tokens.

What “Bag” Means

In a bag, order does not matter: “dog bites man” and “man bites dog” can produce identical vectors if they share the same tokens with the same counts. After Module 9.1 tokenization, BoW simply tallies tokens into vocabulary bins.

Sentence	Tokens	Count vector (dog, bites, man)
dog bites man	dog, bites, man	[1, 1, 1]
man bites dog	man, bites, dog	[1, 1, 1]
dog dog man	dog, dog, man	[2, 0, 1]

Variants

Binary BoW

1 if present, else 0.
Ignores repetition.
Close to summed one-hots with OR.

Count BoW

Raw term frequencies.
CountVectorizer default.
Long docs dominate without normalization.

Weighted BoW

TF-IDF or BM25 weights.
Same geometry, smarter values.
Still orderless (unless n-grams).

Code: CountVectorizer

from sklearn.feature_extraction.text import CountVectorizer

docs = ["dog bites man", "man bites dog", "dog dog man"]
cv = CountVectorizer()
X = cv.fit_transform(docs)
print(cv.get_feature_names_out())
print(X.toarray())
# ['bites' 'dog' 'man']
# [[1 1 1]
#  [1 1 1]
#  [0 2 1]]

# Bigrams restore some order
cv2 = CountVectorizer(ngram_range=(1, 2))
print(cv2.fit_transform(docs).shape)

BoW in the Embedding Timeline

One-hot

Per-token sparse identity.

BoW / TF-IDF

Document histograms & weights.

Word2Vec family

Dense vectors from local context.

nn.Embedding

Task-trained dense tables.

Strengths and Tradeoffs

Strengths

Extremely simple and interpretable dimensions.
Strong baseline for topic/spam/sentiment with linear models.
Cheap to compute and store sparsely.

Tradeoffs

Destroys syntax and most semantics of order.
Vocabulary explosion; sparse high-dimensional space.
No sharing between related words.

Common Misconception

“Bag of words means we only use unordered unique words (a set).” Classical BoW is a multiset: counts matter. Binary BoW is the set-like special case. TF-IDF is BoW with smarter weights, not a different paradigm.

Knowledge Check

Short Answer: What information does BoW discard by design? Answer: Word order / sequence structure (beyond optional n-grams).
True/False: “dog bites man” and “man bites dog” can share the same unigram BoW vector. Answer: True.
Multiple Choice: CountVectorizer produces: (a) dense 300-d embeddings, (b) sparse count features, (c) dependency trees. Answer: (b).
Short Answer: How do bigrams help BoW? Answer: They encode short local phrases, partially restoring order.
True/False: TF-IDF is incompatible with the bag-of-words assumption. Answer: False—it is weighted BoW.
Multiple Choice: Binary BoW stores: (a) frequencies, (b) presence/absence, (c) POS tags. Answer: (b).
Short Answer: Name one task where BoW often remains competitive. Answer: Document classification, spam filtering, or topic detection (any reasonable).
Short Answer: Why is BoW sparse? Answer: Each document uses only a tiny fraction of the vocabulary.
Multiple Choice: Word2Vec differs from BoW mainly by: (a) using GPUs only, (b) learning dense vectors from context windows, (c) requiring one-hot labels. Answer: (b).
True/False: BoW dimensions are typically interpretable as specific words or n-grams. Answer: True.

Key Takeaways

BoW represents text as vocabulary histograms, ignoring order.
Binary, count, and TF-IDF are variants of the same bag idea.
n-grams buy limited local structure at the cost of more features.
BoW is the classical baseline before dense distributional embeddings.
Next: Word2Vec learns continuous word vectors from context.

Trainer’s Guide

Hands-on idea: Have students vectorize a pair of order-sensitive sentences with unigrams vs. bigrams and compare classifier confusion.

Discussion prompt: For which products would identical BoW vectors for “not good” vs. “good” (if “not” were stopped out) be dangerous?

Recap: Bag of words is the orderless count model behind classical NLP features. Next: Word2Vec.