CBOW

How This Lesson Fits the Module & Volume

The Word2Vec overview introduced two sibling architectures. CBOW (Continuous Bag of Words) is the first: given surrounding context tokens, predict the center word. It is “bag-like” inside the window because context embeddings are typically averaged—linking back to the bag-of-words intuition, but now inside a neural embedding model.

This lecture prepares you to contrast CBOW with Skip-gram, which reverses the prediction direction and often handles rare words better.

Learning Objectives

By the end of this lesson, students should be able to:

State the CBOW prediction task: context → center word.
Describe how context vectors are aggregated (usually averaged) inside the window.
Trace a single training example through input embeddings, average, and output projection.
Configure gensim Word2Vec with sg=0 for CBOW.
Compare CBOW’s speed/quality tradeoffs with Skip-gram.
Explain why CBOW behaves like a continuous bag inside a local window.

Definition

CBOW trains word embeddings by maximizing the probability of a target (center) word given the average (or sum) of the embedding vectors of its surrounding context words within a fixed window.

The Prediction Setup

Sentence fragment: the cat sat on the. With window size 2 around sat, context = {the, cat, on, the}. CBOW averages those four context embeddings and predicts “sat” among the vocabulary. Order inside the window is usually ignored—hence “bag of words,” but continuous and local.

1. Window

Collect context tokens around center.

2. Embed

Lookup each context word vector.

3. Aggregate

Average (or sum) context vectors.

4. Predict

Softmax / negative sample → center word.

Parameters and Signal

Piece	Role
Input embedding matrix	Context word → vector (often kept as final embeddings)
Aggregation	Mean of context vectors (order discarded)
Output weights	Score each vocabulary word as the center
Loss	Cross-entropy or negative sampling loss

Code: CBOW with gensim

from gensim.models import Word2Vec

corpus = [
    "neural networks learn embeddings from context".split(),
    "cbow predicts the center word from neighbors".split(),
    "skip gram predicts neighbors from the center".split(),
]
model = Word2Vec(
    corpus,
    vector_size=32,
    window=3,
    min_count=1,
    sg=0,            # CBOW
    negative=5,
    epochs=100,
)
print(model.wv.similarity("center", "neighbors"))
print(model.wv.most_similar("cbow", topn=3))

CBOW vs. Skip-gram (Quick Compare)

Many contexts → one prediction.
Smoothing via averaging.
Often faster; strong on frequent words.

Skip-gram

One center → many context predictions.
More updates per center token.
Often better on rare words.

Shared

Same embedding idea.
Same negative-sampling tricks.
Static type-level vectors.

Strengths and Tradeoffs

Strengths

Efficient: one prediction per window position.
Stable training on frequent tokens.
Simple mental model (average context → guess word).

Tradeoffs

Averaging can wash out rare informative neighbors.
Still ignores order inside the window.
May underperform Skip-gram on sparse vocabulary items.

Common Misconception

“CBOW is the same as document-level bag of words.” Document BoW tallies an entire text into one sparse vector. CBOW uses a local window, continuous embeddings, and a neural prediction loss. The “bag” only refers to unordered aggregation of context embeddings.

Knowledge Check

Short Answer: What does CBOW predict? Answer: The center (target) word from context.
True/False: CBOW typically averages context embeddings before prediction. Answer: True.
Multiple Choice: gensim flag for CBOW is: (a) sg=0, (b) sg=1, (c) cbow=False. Answer: (a).
Short Answer: Why is CBOW called a “bag”? Answer: Context order inside the window is ignored when averaging.
True/False: CBOW usually makes more predictions per center word than Skip-gram. Answer: False—Skip-gram predicts each context word.
Multiple Choice: CBOW tends to be relatively strong on: (a) rare words, (b) frequent words, (c) images. Answer: (b).
Short Answer: Name the two matrices involved conceptually. Answer: Input (context) embeddings and output projection weights.
Short Answer: What window hyperparameter controls? Answer: How many tokens on each side count as context.
Multiple Choice: Negative sampling in CBOW: (a) labels documents, (b) approximates full softmax cheaply, (c) removes stop words. Answer: (b).
True/False: CBOW embeddings remain static after training (one vector per word type). Answer: True.

Key Takeaways

CBOW predicts the center word from averaged context embeddings.
It is efficient and often good for frequent words.
The “bag” is local and continuous—not document-level BoW.
Configure gensim with sg=0 to train CBOW.
Next: reverse the arrow with Skip-gram.

Trainer’s Guide

Hands-on idea: Draw one window on the board, average toy 2-d context vectors, and show which center word would score highest.

Discussion prompt: If one context word is a rare named entity and others are stop-like, what does averaging do to the signal?

Recap: CBOW learns embeddings by guessing the center word from its neighbors. Next: Skip-gram.