Positional Encoding — Module 10.1

How This Lesson Fits the Module & Engineering Practice

Self-attention and multi-head attention mix tokens by content similarity alone. Without an order signal, “dog bites man” and “man bites dog” look interchangeable to a pure attention stack. Positional encoding injects where each token sits in the sequence.

The original transformer used fixed sinusoids; many modern models use learned position embeddings or relative schemes. Either way, positions are not optional for language and most sequence tasks.

Learning Objectives

By the end of this lesson, students should be able to:

Explain why attention alone is permutation-equivariant.
Describe sinusoidal positional encoding and its motivation.
Compare sinusoidal vs learned absolute position embeddings.
Add positions to token embeddings in PyTorch.
State where PE sits relative to encoder/decoder stacks.
Recognize limits of absolute positions (length extrapolation).

Definition

Positional encoding (PE) is a vector (or bias) added to—or otherwise combined with—token representations so the model can distinguish order and relative placement. Without it, bag-of-tokens attention cannot recover sequence order.

Why Attention Needs Positions

Let \(f\) be a self-attention layer (with shared projections). If \(P\) is a permutation matrix that reorders tokens, then—absent position features—

\[f(PX) = P\, f(X)\]

That is permutation equivariance: shuffling inputs only shuffles outputs the same way. Content relationships are preserved; order is not. RNNs carried order implicitly through time; transformers must add it explicitly.

Sinusoidal Positional Encoding

Vaswani et al. defined fixed encodings for position \(\mathrm{pos}\) and dimension \(i\):

\[PE_{(\mathrm{pos}, 2i)} = \sin\!\left(\frac{\mathrm{pos}}{10000^{2i/d}}\right),\quad PE_{(\mathrm{pos}, 2i+1)} = \cos\!\left(\frac{\mathrm{pos}}{10000^{2i/d}}\right)\]

These are added to token embeddings: \(X' = X + PE\). Desired properties include unique patterns per position, smooth interpolation between nearby positions, and a path to relative-offset geometry via linear functions of sinusoids.

Sinusoidal vs Learned

Aspect	Sinusoidal (fixed)	Learned absolute
Parameters	None (formula)	One vector per position index
Training	Frozen PE	Optimized with the model
Length beyond train max	Can evaluate formula further	Often poorly defined / clipped
Flexibility	Inductive bias baked in	Can fit data-specific patterns
Common use	Original transformer paper	Many BERT/GPT-style models

Sinusoidal

Deterministic, no PE params.
Encourages relative structure.
Good teaching / paper baseline.

Learned

Simple nn.Embedding table.
Strong when lengths are bounded.
Watch max-length limits.

Later systems also use relative position biases, RoPE, ALiBi, and other schemes—still solving the same core problem: break pure permutation equivariance in a useful way.

PyTorch: Sinusoidal and Learned PE

import math
import torch
from torch import nn

def sinusoidal_pe(seq_len: int, d_model: int) -> torch.Tensor:
    pe = torch.zeros(seq_len, d_model)
    pos = torch.arange(seq_len).unsqueeze(1).float()
    div = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
    pe[:, 0::2] = torch.sin(pos * div)
    pe[:, 1::2] = torch.cos(pos * div)
    return pe  # (T, d_model)

class LearnedPE(nn.Module):
    def __init__(self, max_len: int, d_model: int):
        super().__init__()
        self.pe = nn.Embedding(max_len, d_model)

    def forward(self, x):
        # x: (B, T, d)
        T = x.size(1)
        positions = torch.arange(T, device=x.device)
        return x + self.pe(positions)

tok = torch.randn(2, 16, 64)
x_sin = tok + sinusoidal_pe(16, 64)
x_learn = LearnedPE(512, 64)(tok)
print(x_sin.shape, x_learn.shape)

What PE enables

Order-sensitive language modeling.
Distinguish subject/object roles by place.
Stable interface with residual stacks.

Limitations

Absolute tables struggle past train length.
Adding PE is not the only modern option.
Bad PE scale can drown token signal.

Common Mistake

Assuming the model “just knows” left-to-right order because text is stored in order in the batch tensor. Storage order does not create features; without PE (or an equivalent bias), attention treats the set of tokens symmetrically.

Misconception

“Positional encoding replaces attention.” No—PE is added (or otherwise fused) so that attention can use position-aware queries and keys. The mixing engine is still attention; PE is the order channel.

Knowledge Check

Short Answer: Why does pure self-attention need positional information? Answer: It is permutation-equivariant and cannot recover token order from content alone.
True/False: Sinusoidal PE uses fixed sin/cos functions of position and dimension. Answer: True.
Multiple Choice: Learned absolute PE is typically stored as: (a) an embedding table, (b) a softmax only, (c) a pooling kernel. Answer: (a).
Short Answer: What does permutation equivariance mean here? Answer: Reordering inputs reorders outputs the same way; no absolute order signal is created.
True/False: PE replaces the need for multi-head attention. Answer: False.
Multiple Choice: A weakness of learned absolute positions: (a) often limited by max training length, (b) they forbid residuals, (c) they remove QKV. Answer: (a).
Short Answer: How are classic sinusoidal encodings combined with tokens? Answer: They are added to the token embeddings.
True/False: RNNs encode order implicitly through recurrence; transformers usually add it explicitly. Answer: True.
Multiple Choice: Which lecture is the natural next block ingredient after PE? (a) Residual Connection, (b) Stemming, (c) Max pooling only. Answer: (a).
Short Answer: Name one alternative to absolute PE used in modern models. Answer: Relative biases, RoPE, ALiBi, or similar relative/rotary schemes.

Key Takeaways

Attention without positions is permutation-equivariant.
Sinusoidal PE is fixed; learned PE is a trainable table.
Both aim to make order available to Q/K/V computation.
Length extrapolation differs sharply between schemes.
Next: Residual Connection.

Trainer’s Guide

Hands-on idea: Train a tiny transformer with and without PE on a order-sensitive toy task (e.g., reverse a short digit string) and compare accuracy.

Discussion prompt: When would you prefer sinusoidal PE over a learned table for a production system?

Recap: Positional encoding supplies order because attention alone is permutation-equivariant. Continue with Residual Connection.