Positional Embedding — Module 10.2

How This Lesson Fits the Module & Volume

Module 10.1 covered fixed sinusoidal positional encoding. Many modern stacks (BERT, GPT, ViT) instead use learned positional embeddings—a trainable vector per absolute index, added to token (or patch) embeddings.

This lecture contrasts encoding vs embedding, shows the PyTorch pattern, and prepares you for Vision Transformer patch positions and Vol. 11 context windows.

Learning Objectives

By the end of this lesson, students should be able to:

Explain why Transformers need positional signals at all.
Contrast learned positional embeddings with sinusoidal positional encodings.
Implement absolute positional embeddings with nn.Embedding.
Discuss max-length limits and extrapolation challenges.
Relate positions to the full-stack input representation.
Preview relative / RoPE-style ideas as later alternatives (conceptually).

Definition

A positional embedding is a learned vector e_pos[i] ∈ R^d_model for sequence index i. It is typically added to the token embedding so the model can distinguish order: h_i = tok_i + e_pos[i]. Unlike sinusoidal encoding, the vectors are parameters updated by gradient descent.

Encoding vs Embedding

Sinusoidal encoding (10.1)

Fixed sin/cos functions of position.
No extra learned position params.
Designed for length extrapolation.

Learned embedding (10.2)

nn.Embedding(max_len, d_model).
Flexible, data-driven positions.
Harder beyond trained max_len.

Shared goal

Break permutation equivariance.
Fuse with token/patch embeddings.
Feed the attention stack order cues.

Aspect	Sinusoidal PE	Learned PE
Parameters	None (formula)	max_len × d_model
Used in	Original Transformer	BERT, GPT-2, ViT (absolute)
Longer than train length	Often better	Needs tricks / relative methods
Implementation	Buffer of sin/cos	`nn.Embedding` lookup

Where Positions Enter the Stack

Token IDs

Lookup tok embeddings.

Position IDs

0..L-1 (or custom).

Add

tok + pos (+ optional type).

Blocks

MHSA / FFN stack.

Code: Learned Absolute Positions

import torch
from torch import nn

class TokenPosEmbedding(nn.Module):
    def __init__(self, vocab_size, d_model, max_len=512, dropout=0.1):
        super().__init__()
        self.tok = nn.Embedding(vocab_size, d_model)
        self.pos = nn.Embedding(max_len, d_model)  # learned positional embedding
        self.drop = nn.Dropout(dropout)
        self.max_len = max_len

    def forward(self, token_ids):
        # token_ids: (B, L)
        B, L = token_ids.shape
        assert L <= self.max_len
        positions = torch.arange(L, device=token_ids.device).unsqueeze(0).expand(B, L)
        x = self.tok(token_ids) + self.pos(positions)
        return self.drop(x)

emb = TokenPosEmbedding(vocab_size=1000, d_model=64, max_len=128)
ids = torch.randint(0, 1000, (2, 20))
print(emb(ids).shape)  # torch.Size([2, 20, 64])
print(emb.pos.weight.shape)  # torch.Size([128, 64]) trainable

Strengths and Tradeoffs

Strengths

Simple and strong within trained lengths.
Easy to add segment/type embeddings (BERT).
Natural fit for fixed patch grids in ViT.

Tradeoffs

Does not extrapolate freely past max_len.
Absolute indices can be brittle under shifts.
Modern LMs often prefer relative / RoPE variants.

Common Misconception

“Positional embedding and positional encoding are interchangeable names for the same algorithm.” In this course, encoding means the fixed sinusoidal scheme from Module 10.1; embedding means a learned lookup table. Both inject order; their parameterizations differ.

Knowledge Check

Short Answer: Why do Transformers need positions? Answer: Self-attention is permutation-equivariant without them.
True/False: Learned positional embeddings are updated by backpropagation. Answer: True.
Multiple Choice: Sinusoidal PE was introduced in: (a) Word2Vec, (b) the 2017 Transformer paper, (c) ResNet. Answer: (b).
Short Answer: How are positional embeddings usually combined with token embeddings? Answer: Element-wise addition (sometimes scaled).
True/False: A learned PE table of size max_len freely handles any longer sequence with no changes. Answer: False—indices beyond max_len are undefined.
Multiple Choice: BERT-style models commonly use: (a) only sinusoidal PE, (b) learned absolute positions (+ segment), (c) no positions. Answer: (b).
Short Answer: Which Module 10.1 lecture covers sinusoidal PE? Answer: Positional Encoding.
Short Answer: Name one reason relative/RoPE methods became popular. Answer: Better length extrapolation / relative distance modeling (any clear reason).
Multiple Choice: In ViT, positional embeddings are added to: (a) raw pixels only, (b) patch (+ class) tokens, (c) the softmax. Answer: (b).
True/False: Encoding vs embedding is a terminology distinction this course maintains on purpose. Answer: True.

Key Takeaways

Positions restore order to attention-based models.
Learned embeddings vs sinusoidal encodings: trainable table vs fixed formula.
Absolute learned PE is simple but length-limited.
Same idea applies to token streams and ViT patches.
Next: Skip Connection.

Trainer’s Guide

Hands-on idea: Train a tiny classifier with and without self.pos; show accuracy collapse when order matters (e.g., reverse-string detection).

Discussion prompt: For a 4k-context chatbot, would you prefer absolute learned PE or a relative scheme—and why?

Recap: Positional embeddings are learned order vectors that complement Module 10.1’s sinusoidal encodings. Continue with Skip Connection.