Word2Vec — Module 9.2

How This Lesson Fits the Module & Volume

Sparse methods—one-hot, BoW, TF-IDF—cannot place “king” near “queen.” Word2Vec (Mikolov et al., 2013) popularized learning dense, low-dimensional vectors so that distributional similarity becomes geometric proximity.

This lecture is the overview of the Word2Vec family. The next two lessons zoom into its architectures: CBOW and Skip-gram. Later lectures compare GloVe and FastText, then move to sentence vectors and trainable nn.Embedding.

Learning Objectives

By the end of this lesson, students should be able to:

State the distributional hypothesis that motivates Word2Vec.
Describe Word2Vec as a shallow neural model that learns dense word vectors from context.
Distinguish the CBOW and Skip-gram training objectives at a high level.
Train or load Word2Vec vectors with gensim and query similarity / analogies.
Explain negative sampling as an efficient softmax approximation.
List strengths and limits of static (non-contextual) embeddings.

Definition

Word2Vec is a family of shallow neural models that learn a dense vector for each vocabulary word by predicting words from their local context (or context from a word). After training, the embedding matrix maps each word ID to a continuous vector (often 100–300 dimensions) usable as features.

Distributional Hypothesis

Words that occur in similar contexts tend to have similar meanings. Word2Vec operationalizes this: if “coffee” and “tea” both appear near “cup,” “drink,” and “hot,” their vectors are pulled closer during training.

One-hot / BoW

Sparse, no similarity.

Word2Vec

Dense vectors from local context.

GloVe / FastText

Global stats & subwords.

Contextual

Transformers (Vol. 10).

Two Architectures (Preview)

CBOW

Context words → predict center.
Faster on frequent words.
Detail: CBOW lecture.

Skip-gram

Center word → predict context.
Often better on rare words.
Detail: Skip-gram lecture.

Shared idea

Shallow net, large corpus.
Embedding = learned weight row.
Negative sampling for speed.

Code: gensim Word2Vec

from gensim.models import Word2Vec

sentences = [
    ["the", "cat", "sat", "on", "the", "mat"],
    ["the", "dog", "sat", "on", "the", "log"],
    ["cats", "and", "dogs", "are", "pets"],
]
model = Word2Vec(
    sentences,
    vector_size=50,
    window=2,
    min_count=1,
    sg=1,              # 1 = Skip-gram, 0 = CBOW
    negative=5,
    epochs=50,
)
print(model.wv.most_similar("cat", topn=3))
print(model.wv["dog"].shape)   # (50,)

What You Can Do with the Vectors

Operation	Example	Interpretation
Similarity	cos(cat, dog)	Relatedness in embedding space
Analogy	king - man + woman ≈ queen	Linear regularities (approximate)
Features	Average word vectors	Simple document embedding
Init	Load into nn.Embedding	Warm-start neural NLP models

Strengths and Tradeoffs

Strengths

Dense, reusable features across tasks.
Captures similarity and some analogies.
Scales to large corpora with negative sampling.

Tradeoffs

Static: one vector per word type (not per sense/context).
“bank” (finance vs. river) shares one embedding.
Needs enough data; poor on rare morphology without FastText.

Common Misconception

“Word2Vec understands meaning like a language model.” It optimizes a local co-occurrence prediction objective. Vectors encode distributional patterns, not grounded world knowledge. Contextual models in Volume 10 go further by producing different vectors for the same word in different sentences.

Knowledge Check

Short Answer: State the distributional hypothesis in one sentence. Answer: Words in similar contexts tend to have similar meanings.
True/False: Word2Vec produces sparse V-dimensional one-hot vectors. Answer: False—it produces dense low-dimensional vectors.
Multiple Choice: In gensim, sg=1 selects: (a) CBOW, (b) Skip-gram, (c) GloVe. Answer: (b).
Short Answer: Name Word2Vec’s two main architectures. Answer: CBOW and Skip-gram.
True/False: Negative sampling avoids a full softmax over the vocabulary. Answer: True.
Multiple Choice: A limitation of Word2Vec is: (a) it cannot run on CPU, (b) one vector per word type, (c) it requires labeled data. Answer: (b).
Short Answer: What classic analogy is often cited for embeddings? Answer: king - man + woman ≈ queen (or similar).
Short Answer: How can Word2Vec vectors initialize a neural NLP model? Answer: Copy them into an nn.Embedding weight matrix.
Multiple Choice: Word2Vec training mainly uses: (a) document labels, (b) local context windows, (c) dependency trees only. Answer: (b).
True/False: Static Word2Vec vectors change with sentence context at inference time. Answer: False.

Key Takeaways

Word2Vec learns dense vectors so distributional similarity becomes geometry.
CBOW and Skip-gram are the two core training setups.
gensim provides a practical API for training and similarity queries.
Embeddings are static and sense-agnostic—a limit later lectures address.
Next: dive into CBOW.

Trainer’s Guide

Hands-on idea: Train tiny Word2Vec on a toy corpus; compare most_similar before/after more epochs or a larger window.

Discussion prompt: Why might averaging Word2Vec vectors be a weak sentence representation for negation-heavy text?

Recap: Word2Vec turns co-occurrence prediction into dense semantic vectors. Continue with CBOW.