Encoder — Module 10.1

How This Lesson Fits the Module & Volume

Volume 09 closed with the embedding layer: token IDs become dense vectors via nn.Embedding. Those vectors are still static per token—they do not yet know sentence context. Volume 08 showed how RNNs and GRUs build context sequentially, but that path creates a bottleneck: every token must wait for the previous one, and a single final hidden state is asked to summarize an entire sequence.

Volume 10 opens the Attention & Transformers arc. The encoder is the first architectural idea: a stack that reads the full source sequence and emits a rich contextual representation for every position. Later lectures unpack the machinery inside—Query, Key, Value, and Attention—but the encoder is the map of where that machinery lives.

Learning Objectives

By the end of this lesson, students should be able to:

Define an encoder as a sequence-to-sequence contextualizer that maps tokens to contextual vectors.
Contrast RNN/GRU sequential encoding with parallel, attention-based encoding.
Explain the information bottleneck of a single final hidden state in classic seq2seq.
Sketch the Transformer encoder block: self-attention, residuals, layer norm, and FFN.
Build a minimal encoder stack in PyTorch that turns embeddings into contextual hidden states.
State what an encoder produces and how a decoder will consume it.

Definition

An encoder is a neural module that maps an input sequence x₁, …, x_n to a sequence of contextual representations h₁, …, h_n (or a pooled vector derived from them). In attention-based models, each h_i can depend on every other token in the input—not only on tokens to its left.

From Embeddings to Context

An embedding table gives each word a fixed vector. The sentence “the bank by the river” and “the bank raised rates” share the same embedding for bank until a contextualizer mixes neighboring information. The encoder is that contextualizer: it takes embedded tokens (plus, later, positional encoding) and produces representations that resolve meaning in context.

1. Tokens

IDs from the vocabulary.

2. Embed

nn.Embedding lookup.

3. Encode

Stack of attention + FFN blocks.

4. Memory

Contextual h_1..n for the decoder.

The RNN Bottleneck We Are Leaving Behind

Classic encoder–decoder RNNs compress the source into one vector (or a short chain of states). Long sentences lose early details; translation and summarization suffer. Attention was invented precisely so the decoder could look back at all encoder states—and Transformers go further by making the encoder itself fully parallel and attention-based.

Property	RNN / GRU Encoder	Attention Encoder
Computation order	Left-to-right (or bidirectional)	All positions in parallel
Path length between tokens	O(n) steps	O(1) via attention
Output	Often one final state	Full sequence of states
Long-range deps	Fragile (vanishing signal)	Direct pairwise links
GPU utilization	Limited by sequential steps	Highly parallel matrix ops

What Lives Inside a Modern Encoder

Self-Attention

Each position attends to all others.
Builds contextual mixture of values.
See Self Attention.

Feed-Forward

Position-wise MLP after attention.
Adds capacity per token.
See FFN.

Stability Glue

Residuals + LayerNorm.
Deep stacks train reliably.
Repeated N times (e.g. 6, 12).

Minimal Encoder Skeleton in PyTorch

This toy encoder embeds tokens and runs a few Transformer encoder layers. Real models add positional encodings and masks; the shape story is what matters now.

import torch
from torch import nn

class TinyEncoder(nn.Module):
    def __init__(self, vocab_size, d_model=64, nhead=4, num_layers=2):
        super().__init__()
        self.emb = nn.Embedding(vocab_size, d_model)
        layer = nn.TransformerEncoderLayer(
            d_model=d_model, nhead=nhead,
            dim_feedforward=4 * d_model, batch_first=True,
        )
        self.encoder = nn.TransformerEncoder(layer, num_layers=num_layers)

    def forward(self, token_ids, src_key_padding_mask=None):
        # token_ids: (batch, seq_len)
        x = self.emb(token_ids)                          # (B, T, d_model)
        return self.encoder(x, src_key_padding_mask=src_key_padding_mask)

enc = TinyEncoder(vocab_size=1000)
ids = torch.randint(0, 1000, (2, 12))
memory = enc(ids)
print(memory.shape)   # torch.Size([2, 12, 64])  — one vector per token

Encoder Outputs = Memory for Downstream Work

In translation, the encoder’s memory tensor is what the decoder reads via cross-attention. In classification, you may pool encoder states (CLS token or mean) and feed a linear head. Either way, the encoder’s job is representation—not generation.

Strengths

Full-sequence context at every position.
Parallelizable training and inference over length.
Reusable memory for many decoder tasks.

Tradeoffs

Quadratic cost in sequence length for dense attention.
Needs positional signals (order is not inherent).
More parameters than a shallow RNN for tiny data.

Common Misconception

“The encoder outputs one vector like an RNN.” In Transformers, the encoder outputs a sequence of vectors—one contextual state per input token. Pooling to a single vector is an optional downstream choice, not the encoder’s definition.

Knowledge Check

Short Answer: What does an encoder map an input sequence to? Answer: A sequence of contextual representations (one per position), or a pooled vector derived from them.
True/False: Volume 09 embeddings already provide full sentence context for each token. Answer: False—static embeddings lack context until an encoder (or similar) mixes information.
Multiple Choice: The classic RNN seq2seq bottleneck is: (a) too many GPUs, (b) compressing the whole source into one final state, (c) using ReLU. Answer: (b).
Short Answer: Name two components inside a Transformer encoder block. Answer: Self-attention and a feed-forward network (plus residuals/LayerNorm).
True/False: Attention-based encoders can process all positions in parallel. Answer: True.
Multiple Choice: In the PyTorch sketch, memory.shape for batch 2, length 12, d_model 64 is: (a) (2, 64), (b) (2, 12, 64), (c) (12, 2, 64). Answer: (b).
Short Answer: Why do encoders need positional encoding later? Answer: Self-attention is permutation-equivariant; without positions, order is invisible.
True/False: The decoder generates tokens; the encoder’s primary role is representation. Answer: True.
Multiple Choice: Path length between distant tokens in self-attention is: (a) O(n), (b) O(1), (c) O(log n) only. Answer: (b).
Short Answer: What module consumes encoder memory in translation? Answer: The decoder (via cross-attention).

Key Takeaways

Encoders turn embedded tokens into contextual hidden states for every position.
They replace the RNN “single vector” bottleneck with a full memory sequence.
A Transformer encoder stacks self-attention, FFN, residuals, and LayerNorm.
Next: the Decoder learns to read that memory and generate outputs.

Trainer’s Guide

Hands-on idea: Run TinyEncoder, print shapes, then add a padding mask and show that PAD positions can be ignored.

Discussion prompt: Compare translating a 50-token sentence with a single GRU state versus a 50-vector encoder memory—what information survives?

Recap: The encoder contextualizes embeddings into a parallel sequence of states. Continue with Decoder.