Embedding Layer — Module 9.2

How This Lesson Fits the Module & Volume

Module 9.2 began at one-hot encoding—bridging Module 9.1 linguistic preprocessing to numbers—then climbed through TF-IDF, BoW, Word2Vec (CBOW / Skip-gram), GloVe, FastText, and sentence embeddings.

This capstone brings embeddings into the PyTorch training loop: nn.Embedding is a learnable lookup table. It is how modern NLP models—RNNs from Volume 08 and especially Attention & Transformers in Volume 10—turn token IDs into differentiable dense vectors end-to-end.

Learning Objectives

By the end of this lesson, students should be able to:

Explain nn.Embedding as a trainable weight matrix indexed by token IDs.
Relate embedding lookup to a multiply by a one-hot vector (without materializing it).
Build a tiny classifier that embeds tokens, pools, and predicts a label in PyTorch.
Initialize an embedding layer from pretrained Word2Vec/GloVe/FastText weights.
Choose padding_idx, freezing vs. fine-tuning, and embedding dimension thoughtfully.
Connect this module’s static methods to contextual embeddings coming in Volume 10.

Definition

An embedding layer (torch.nn.Embedding) stores a matrix W ∈ R^V×d and maps integer indices to rows of W. During backpropagation, only the rows touched by the batch receive gradient updates—learning task-specific dense representations.

Full Module Arc

One-hot

Sparse identity vectors.

BoW / TF-IDF

Sparse document features.

W2V / GloVe / FT

Pretrained dense static vectors.

nn.Embedding

Task-trained (or fine-tuned) table.

Lookup = Efficient One-Hot Multiply

If e_i is a one-hot vector for index i, then e_i^T W selects row i. nn.Embedding performs that selection directly. This is why embeddings replaced giant one-hot inputs in neural NLP: same information, dense trainable geometry, vastly less memory.

API / idea	Meaning
`nn.Embedding(V, d)`	V vocabulary rows, d-dimensional vectors
`padding_idx`	Row kept at zeros; no gradient (for PAD)
`weight`	The `(V, d)` parameter matrix
`from_pretrained`	Load GloVe/Word2Vec rows; optional freeze

Code: Classifier with nn.Embedding

import torch
from torch import nn

class BagClassifier(nn.Module):
    def __init__(self, vocab_size, embed_dim, num_classes, pad_idx=0):
        super().__init__()
        self.emb = nn.Embedding(vocab_size, embed_dim, padding_idx=pad_idx)
        self.fc = nn.Linear(embed_dim, num_classes)

    def forward(self, token_ids):
        # token_ids: (batch, seq_len)
        x = self.emb(token_ids)          # (batch, seq_len, embed_dim)
        mask = (token_ids != self.emb.padding_idx).unsqueeze(-1).float()
        pooled = (x * mask).sum(1) / mask.sum(1).clamp(min=1e-9)
        return self.fc(pooled)

model = BagClassifier(vocab_size=1000, embed_dim=64, num_classes=2)
batch = torch.randint(1, 1000, (4, 12))  # fake token ids
batch[:, -2:] = 0                        # pad
print(model(batch).shape)                # torch.Size([4, 2])

# Optional: warm-start from pretrained rows
# model.emb = nn.Embedding.from_pretrained(pretrained_tensor, freeze=False)

Pretrained vs. From Scratch

From scratch

Random init, learn on task data.
Needs enough labels.
Fully task-specific geometry.

Frozen pretrained

Load GloVe/W2V/FastText.
Train only the head.
Good when data is scarce.

Fine-tuned

Load then unfreeze.
Adapt to domain jargon.
Watch for overfitting.

Strengths and Tradeoffs

Strengths

End-to-end differentiable with any PyTorch model.
Memory-efficient vs. explicit one-hots.
Easy warm-start from Module 9.2 static vectors.

Tradeoffs

Still one vector per token ID (type-level) unless the rest of the net is contextual.
Vocab / OOV decisions remain critical.
Deep context needs attention stacks (Vol. 10).

Common Misconception

“nn.Embedding is the same as Word2Vec.” Word2Vec is a pretraining objective on unlabeled text. nn.Embedding is a layer: a parameter table updated by whatever loss you attach (classification, LM, etc.). You can initialize the layer with Word2Vec weights, then fine-tune—or learn it entirely from your supervised objective.

Looking Ahead: Volume 10

Embedding layers feed every modern sequence model. In Volume 10 you will stack attention on top of token embeddings so each position’s vector becomes contextual—different for “bank” in finance vs. river contexts. Module 9.2 gave you the representation toolkit; Attention & Transformers put it to work at scale.

Knowledge Check

Short Answer: What does nn.Embedding(V, d) store? Answer: A V×d trainable weight matrix (one vector per token ID).
True/False: Embedding lookup is equivalent to multiplying by a one-hot vector. Answer: True.
Multiple Choice: padding_idx is used to: (a) delete the vocab, (b) zero a PAD row and block its gradient, (c) enable TF-IDF. Answer: (b).
Short Answer: How do you load GloVe rows into PyTorch? Answer: nn.Embedding.from_pretrained(...) or copy into .weight.
True/False: nn.Embedding always produces contextualized vectors by itself. Answer: False—context comes from later layers.
Multiple Choice: Freezing pretrained embeddings means: (a) deleting them, (b) not updating those weights, (c) converting to one-hot. Answer: (b).
Short Answer: Name one reason embeddings beat raw one-hots in neural nets. Answer: Dense, trainable, memory-efficient similarity structure.
Short Answer: What Volume 10 topic builds on token embeddings for context? Answer: Attention (and Transformers).
Multiple Choice: In the sample classifier, pooling happens: (a) before embedding, (b) after embedding over the sequence, (c) only in sklearn. Answer: (b).
True/False: Word2Vec is an objective; nn.Embedding is a layer that can hold its (or other) vectors. Answer: True.

Key Takeaways

nn.Embedding is the PyTorch home for learnable token vectors.
It efficiently replaces one-hot × matrix multiplies in the forward pass.
Warm-start from Word2Vec/GloVe/FastText or train from scratch.
Module 9.2’s path: sparse counts → static dense → trainable layers.
Next volume: Attention makes embeddings contextual.

Trainer’s Guide

Hands-on idea: Train BagClassifier on a tiny sentiment set; compare random embeddings vs. frozen GloVe init on the same split.

Discussion prompt: Capstone review—for a new text project, when do you stop at TF-IDF, when load FastText, and when train nn.Embedding inside a net?

Recap: The embedding layer turns token IDs into trainable dense vectors and closes Volume 09’s representation module. Continue to Vol. 10 Attention.