Sentence Embeddings — Module 9.2

How This Lesson Fits the Module & Volume

So far Module 9.2 produced vectors for words: from one-hot through Word2Vec, GloVe, and FastText. Many tasks need a single vector for an entire sentence or short paragraph—semantic search, clustering, duplicate detection.

Sentence embeddings aggregate or encode token sequences into fixed-length vectors. Simple baselines average word vectors; stronger methods use trained encoders (and, looking ahead to Volume 10, transformers). This lecture bridges word-level geometry to sequence-level representation before the capstone embedding layer.

Learning Objectives

By the end of this lesson, students should be able to:

Define a sentence embedding as a fixed-size vector for a variable-length text.
Implement mean-pooled Word2Vec/GloVe sentence vectors in NumPy/PyTorch.
Explain limitations of averaging (negation, word order, polysemy).
Contrast bag-averaged embeddings with encoder-based sentence models.
Use cosine similarity for sentence retrieval as a worked pattern.
Preview how Volume 10 attention/transformers produce contextual sentence vectors.

Definition

A sentence embedding is a dense vector representation of a sentence (or short document) in a fixed dimension d, designed so that semantically similar sentences lie nearby under a similarity metric such as cosine similarity.

From Word Vectors to Sentence Vectors

After tokenization, look up each token’s embedding and reduce the sequence. The simplest reduction is the arithmetic mean. It is surprisingly strong as a baseline—and clearly broken for order-sensitive meaning.

1. Tokenize

Module 9.1 pipeline.

2. Embed tokens

Word2Vec / GloVe / FastText / nn.Embedding.

3. Pool

Mean, max, or weighted pool.

4. Compare

Cosine similarity / classifiers.

Code: Mean-Pool Static Embeddings

import numpy as np
from gensim.models import Word2Vec

corpus = [
    "the movie was fantastic and fun".split(),
    "the film was awful and boring".split(),
    "fantastic fun movie night".split(),
]
w2v = Word2Vec(corpus, vector_size=32, window=2, min_count=1, sg=1, epochs=100)

def sentence_vec(tokens, model):
    vecs = [model.wv[t] for t in tokens if t in model.wv]
    return np.mean(vecs, axis=0)

def cosine(a, b):
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

s0 = sentence_vec(corpus[0], w2v)
s1 = sentence_vec(corpus[1], w2v)
s2 = sentence_vec(corpus[2], w2v)
print(cosine(s0, s2), cosine(s0, s1))  # similar sentiment closer?

Pooling Choices

Mean pool

Simple, stable baseline.
Washes out word order.
Sensitive to length mix.

Max pool

Keeps salient dimensions.
Can be noisy.
Ignores order too.

Encoder pool

RNN/Transformer [CLS]/mean.
Context-aware tokens first.
Vol. 10 territory.

When Averaging Fails

Phenomenon	Example	Why mean-pool struggles
Negation	“not good” vs “good”	Vectors may still be close if “not” is weak
Order	“dog bites man”	Same bag as reverse
Polysemy	“bank” senses	Static word vector is mixed
Composition	Idioms	Meaning ≠ average of parts

Strengths and Tradeoffs

Strengths

Fixed-size features for any sentence length.
Mean-pool baselines are cheap and often useful.
Natural input to clustering and semantic search.

Tradeoffs

Bag pooling loses syntax.
Quality capped by underlying word vectors.
True SOTA needs contextual encoders (Vol. 10).

Common Misconception

“A sentence embedding is just concatenating all word vectors.” Concatenation grows with sentence length and breaks fixed-size model heads. Embeddings for sentences are fixed-dimension summaries—via pooling or a dedicated encoder—not variable-length concatenations.

Knowledge Check

Short Answer: What is a sentence embedding? Answer: A fixed-size dense vector representing a whole sentence.
True/False: Mean-pooling word vectors preserves full word order. Answer: False.
Multiple Choice: A common similarity for sentence vectors is: (a) edit distance on one-hots, (b) cosine similarity, (c) BLEU only. Answer: (b).
Short Answer: Give one failure mode of averaging embeddings. Answer: Negation, order, polysemy, or idioms (any).
True/False: Sentence embeddings must have dimension equal to vocabulary size. Answer: False.
Multiple Choice: Compared with BoW, mean-pooled Word2Vec sentences: (a) are denser and capture some similarity, (b) are always sparse, (c) require labels. Answer: (a).
Short Answer: What does Volume 10 add for stronger sentence vectors? Answer: Attention / transformers (contextual encoders).
Short Answer: Why not concatenate all token vectors? Answer: Length varies; models expect fixed-size inputs.
Multiple Choice: Max pooling over word vectors: (a) keeps order, (b) selects salient feature dimensions, (c) trains GloVe. Answer: (b).
True/False: You can build a simple sentence embedder from pretrained GloVe + mean pool. Answer: True.

Key Takeaways

Sentence embeddings map variable-length text to fixed dense vectors.
Mean-pooling static word vectors is the essential baseline.
Order, negation, and sense still demand contextual encoders.
Cosine similarity enables retrieval and clustering demos.
Capstone next: the trainable Embedding Layer in PyTorch.

Trainer’s Guide

Hands-on idea: Build a 5-sentence mini search engine with mean-pooled vectors; plant a negation pair and watch ranking fail.

Discussion prompt: For FAQ matching, when is averaged GloVe enough vs. when do you need a sentence-transformer-style model?

Recap: Sentence embeddings summarize whole utterances as fixed vectors. Finish Module 9.2 with the Embedding Layer.