← Master Index
Vol. 09 Module 9.2 Lecture

GloVe

Word Embeddings

How This Lesson Fits the Module & Volume

Word2Vec (via CBOW / Skip-gram) learns from local streaming windows. GloVe (Global Vectors, Pennington et al., 2014) instead factors a global word–word co-occurrence matrix: it blends count-based statistics (spirit of BoW/TF-IDF corpora) with the dense geometry of neural embeddings.

In the module timeline, GloVe is the “global statistics” cousin of Word2Vec. Next, FastText adds subword information for morphology and OOV robustness.

Learning Objectives

By the end of this lesson, students should be able to:

  • Explain GloVe as learning embeddings from global co-occurrence ratios.
  • Contrast window-streaming Word2Vec with matrix-based GloVe.
  • Interpret the idea that vector differences encode co-occurrence ratios.
  • Load pretrained GloVe vectors (e.g. via gensim or torchtext-style workflows) and query similarity.
  • List typical pretrained GloVe corpora/dimensions (e.g. 50d–300d on Wikipedia+Gigaword).
  • Place GloVe among static embedding methods before FastText and contextual models.
Definition

GloVe trains word vectors so that the dot product of two word embeddings approximates the logarithm of their co-occurrence count (with biases). Training minimizes a weighted least-squares loss over nonzero entries of a global co-occurrence matrix built from a large corpus.

Global Counts, Dense Vectors

First, scan the corpus once and count how often word j appears in the context of word i. That yields a sparse matrix X. GloVe then learns vectors wi, wj and biases such that wi·wj + bi + bj ≈ log Xij. Frequent co-occurrences get higher loss weight, but not so high that they dominate.

1. Count

Build global co-occurrence matrix X.

2. Weight

Downweight very rare and extremely frequent pairs.

3. Factor

Fit embeddings to log co-occurrence.

4. Use

Similarity, analogies, model init.

Word2Vec vs. GloVe

Word2Vec

  • Local window predictions.
  • Online / streaming friendly.
  • Implicitly uses co-occurrence.

GloVe

  • Explicit global matrix.
  • Weighted least squares.
  • Leverages corpus-wide ratios.

Shared outcome

  • Dense static vectors.
  • Linear analogy structure.
  • No context sensitivity.

Code: Using Pretrained GloVe with gensim

from gensim.models import KeyedVectors import gensim.downloader as api # Downloads a pretrained GloVe set (first run caches locally) glove = api.load("glove-wiki-gigaword-50") # 50-dimensional vectors print(glove.most_similar("computer", topn=5)) print(glove.similarity("king", "queen")) # Classic analogy: king - man + woman print(glove.most_similar(positive=["king", "woman"], negative=["man"], topn=3)) # Map into a PyTorch embedding row later via glove[word]

Strengths and Tradeoffs

Strengths

  • Uses global statistics, not only local samples.
  • Strong pretrained checkpoints widely available.
  • Competitive with Word2Vec on many analogy/similarity suites.

Tradeoffs

  • Building X for huge corpora is memory-heavy.
  • Still one vector per word type.
  • OOV words need <UNK> or fallback (FastText helps).
Common Misconception

“GloVe is just TF-IDF with fewer dimensions.” TF-IDF weights document–term matrices for sparse retrieval/classification. GloVe factorizes word–word co-occurrence into dense semantic vectors. Related count spirit, different object and goal.

Knowledge Check

  1. Short Answer: What does GloVe stand for? Answer: Global Vectors (for word representation).
  2. True/False: GloVe primarily trains from a global co-occurrence matrix. Answer: True.
  3. Multiple Choice: GloVe’s loss encourages wi·wj to match: (a) POS tags, (b) log co-occurrence, (c) TF-IDF of documents. Answer: (b).
  4. Short Answer: How does GloVe differ from Skip-gram at a high level? Answer: Global matrix factorization vs. local window prediction (streaming).
  5. True/False: Pretrained GloVe vectors are contextual (change per sentence). Answer: False.
  6. Multiple Choice: A common gensim loader call uses: (a) api.load("glove-wiki-gigaword-50"), (b) TfidfVectorizer, (c) nn.LSTM. Answer: (a).
  7. Short Answer: Name one limitation GloVe shares with Word2Vec. Answer: Static embeddings / one vector per type / OOV issues (any).
  8. Short Answer: What matrix must be built before GloVe training? Answer: The word–word co-occurrence matrix.
  9. Multiple Choice: Next method that adds subword info is: (a) one-hot, (b) FastText, (c) average pooling only. Answer: (b).
  10. True/False: GloVe and Word2Vec both aim for dense semantic geometry. Answer: True.

Key Takeaways

  • GloVe learns dense vectors by factorizing global co-occurrence counts.
  • It complements Word2Vec’s local predictive training.
  • Pretrained GloVe checkpoints are easy to load and reuse.
  • Like Word2Vec, vectors are static and type-level.
  • Next: FastText for subword embeddings.
Trainer’s Guide

Hands-on idea: Load glove-wiki-gigaword-50 and run three analogies from your domain (e.g. city–country, company–CEO if present).

Discussion prompt: When would rebuilding a domain-specific co-occurrence matrix beat downloading generic GloVe?

Recap: GloVe turns global co-occurrence statistics into dense word vectors. Next: FastText.