Skip Gram — Module 9.2

How This Lesson Fits the Module & Volume

After CBOW, we flip the Word2Vec objective. Skip-gram takes the center word and predicts each context word in the window. That extra supervision per token often improves rare-word vectors and is the default many practitioners reach for first.

Together, CBOW and Skip-gram complete the classic Word2Vec pair. Next, GloVe attacks the same co-occurrence idea with a global matrix factorization style objective.

Learning Objectives

By the end of this lesson, students should be able to:

State the Skip-gram task: center word → context words.
Explain why Skip-gram creates more training pairs per window than CBOW.
Describe negative sampling as used in Skip-gram training.
Train Skip-gram embeddings in gensim (sg=1) and inspect neighbors.
Reason about when Skip-gram outperforms CBOW (rare words, smaller data).
Connect Skip-gram embeddings to downstream use via averaging or nn.Embedding init.

Definition

Skip-gram learns embeddings by maximizing the probability of context words within a window given a single center (input) word. Each (center, context) pair is a training example; negative sampling contrasts true pairs against random noise words.

Center → Context

For center sat with window 2 in the cat sat on the, Skip-gram forms pairs: (sat, the), (sat, cat), (sat, on), (sat, the). The model pushes the embedding of “sat” to be predictive of those neighbors—and unpredicative of randomly drawn negatives.

1. Pick center

Current token in the stream.

2. Form pairs

Each context token in the window.

3. Score

Dot product of center & context vectors.

4. Negatives

Push down scores for noise words.

Why More Updates Help Rare Words

Aspect	CBOW	Skip-gram
Prediction direction	Context → center	Center → context
Examples per window	~1	~2 × window
Rare center words	Averaged away by context	Directly used as input
Typical speed	Faster	Slower (more pairs)

Code: Skip-gram + Negative Sampling

from gensim.models import Word2Vec

sentences = [
    ["rare", "words", "need", "more", "signal"],
    ["skip", "gram", "predicts", "context", "from", "center"],
    ["negative", "sampling", "speeds", "up", "training"],
]
model = Word2Vec(
    sentences,
    vector_size=64,
    window=2,
    min_count=1,
    sg=1,            # Skip-gram
    negative=10,     # noise samples per positive
    epochs=80,
)
print(model.wv.most_similar("skip", topn=3))

# Export for PyTorch warm-start later
import numpy as np
vectors = np.array([model.wv[w] for w in model.wv.index_to_key])
print(vectors.shape)

Negative Sampling Intuition

Full Softmax

Normalize over all V words.
Prohibitive for large V.
Exact but slow.

Negative Sampling

True pair vs. k noise pairs.
Logistic losses on dots.
Default practical choice.

Result

Vectors encode local co-occurrence.
Similar contexts → close vectors.
Ready for similarity queries.

Strengths and Tradeoffs

Strengths

Strong rare-word embeddings.
Flexible window / negative settings.
Industry-standard static baseline.

Tradeoffs

More compute than CBOW per token.
Still one vector per type (no context).
Window size is a coarse syntax proxy.

Common Misconception

“Skip-gram skips words in the sentence as a preprocessing step.” The name refers to predicting words that may be several positions away (skipping intermediates in the window), not deleting tokens from the corpus. Training still uses the full token stream.

Knowledge Check

Short Answer: What does Skip-gram predict? Answer: Context words given the center word.
True/False: Skip-gram typically generates more training pairs per window than CBOW. Answer: True.
Multiple Choice: gensim Skip-gram uses: (a) sg=1, (b) sg=0, (c) skip=True. Answer: (a).
Short Answer: Why can Skip-gram help rare words? Answer: The rare word is used as the input/center, with multiple context predictions.
True/False: Negative sampling compares true context words against random noise words. Answer: True.
Multiple Choice: Relative to CBOW, Skip-gram is often: (a) faster, (b) slower but better on rares, (c) identical. Answer: (b).
Short Answer: Name one hyperparameter that sets noise examples per positive. Answer: negative (k).
Short Answer: Does Skip-gram produce contextual embeddings at inference? Answer: No—still one static vector per word type.
Multiple Choice: After Skip-gram, a natural next classical method is: (a) GloVe, (b) k-means on pixels, (c) max-pooling only. Answer: (a).
True/False: The Skip-gram name means the trainer deletes every other sentence. Answer: False.

Key Takeaways

Skip-gram predicts context from the center word.
More pairs per window often improve rare-word quality.
Negative sampling makes large-vocabulary training practical.
Vectors remain static; contextual models come in Vol. 10.
Next: GloVe learns from global co-occurrence statistics.

Trainer’s Guide

Hands-on idea: List all Skip-gram pairs for a 7-token sentence with window=2; count how many CBOW predictions the same windows would yield.

Discussion prompt: For a domain with many rare technical terms, would you start with CBOW or Skip-gram—and why?

Recap: Skip-gram learns by predicting neighbors from the center word. Continue with GloVe.