Transformer — Module 10.2

How This Lesson Fits the Module & Volume

After the history, this lecture defines what a Transformer is: a stack of attention + FFN blocks with residuals and normalization. It sits between the story and the full-stack walkthrough.

Every Module 10.1 concept—self-attention, MHA, positions, LayerNorm, FFN, residuals—reappears here as an architectural contract.

Learning Objectives

By the end of this lesson, students should be able to:

Define the Transformer as stacked attention–FFN blocks with skip connections and LayerNorm.
Contrast encoder–decoder, encoder-only, and decoder-only configurations.
List the roles of embeddings, positional signals, and the output head.
Implement a minimal Transformer encoder layer skeleton in PyTorch.
Relate d_model, nhead, and num_layers to model capacity.
Point to where Module 10.2 will zoom into each subcomponent.

Definition

A Transformer is a deep network whose primary layer is: multi-head (self-)attention, a residual add & normalize, a position-wise feed-forward network, and another residual add & normalize—repeated N times over token embeddings augmented with position information.

The Contract of One Layer

Tokens in

Embeddings + positions.

Mix

Multi-head attention.

Transform

Position-wise FFN.

Stabilize

Residuals + LayerNorm.

Attention mixes information across positions; the FFN transforms each position independently (see Feed Forward Network). Residuals keep an identity path (see Residual Connection and later Skip Connection). LayerNorm keeps activations well-scaled.

Hyperparameter	Meaning
`d_model`	Width of token representations
`nhead`	Number of attention heads
`dim_feedforward`	Hidden size inside the FFN (often 4×d_model)
`num_layers`	How many identical blocks are stacked
`dropout`	Regularization on attention/FFN paths

Three Common Layouts

Encoder–Decoder

Original NMT Transformer.
Cross-attention links stacks.
See encoder/decoder blocks next.

Encoder-only

BERT-style bidirectional.
Classification / NLU heads.
No causal mask required.

Decoder-only

GPT-style autoregressive.
Masked attention.
Bridge to Vol. 11 LLMs.

Code: Minimal Encoder Layer

import torch
from torch import nn

class TinyTransformerEncoderLayer(nn.Module):
    def __init__(self, d_model=64, nhead=4, dim_ff=128, dropout=0.1):
        super().__init__()
        self.self_attn = nn.MultiheadAttention(d_model, nhead, dropout=dropout, batch_first=True)
        self.ff = nn.Sequential(
            nn.Linear(d_model, dim_ff),
            nn.ReLU(),
            nn.Linear(dim_ff, d_model),
        )
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        self.drop = nn.Dropout(dropout)

    def forward(self, x, key_padding_mask=None):
        # Pre-norm style (common in modern stacks)
        h = self.norm1(x)
        attn_out, _ = self.self_attn(h, h, h, key_padding_mask=key_padding_mask)
        x = x + self.drop(attn_out)
        h = self.norm2(x)
        x = x + self.drop(self.ff(h))
        return x

layer = TinyTransformerEncoderLayer()
tokens = torch.randn(2, 10, 64)  # (batch, seq, d_model)
print(layer(tokens).shape)       # torch.Size([2, 10, 64])

Strengths and Tradeoffs

Strengths

Uniform, scalable stack of identical blocks.
Strong inductive bias for pairwise interactions.
One recipe transfers across modalities.

Tradeoffs

Quadratic cost in sequence length.
Needs explicit position handling.
Hyperparameter surface (heads, depth, width) is large.

Common Misconception

“nn.Transformer is the only way to build one.” PyTorch’s module is convenient, but production models often hand-roll layers (pre-norm, custom masks, fused kernels). Understanding the block matters more than memorizing one API.

Where This Goes Next

The full-stack lecture wires embeddings → PE → stacked blocks → output head. Then encoder and decoder blocks unpack the internals.

Knowledge Check

Short Answer: Name the two main sublayers inside a Transformer block. Answer: Multi-head attention and the position-wise FFN.
True/False: The FFN mixes information across different sequence positions. Answer: False—it is applied independently per position.
Multiple Choice: d_model is: (a) vocabulary size, (b) representation width, (c) number of GPUs. Answer: (b).
Short Answer: What do residual connections provide in each block? Answer: An identity shortcut around attention/FFN for stable deep training.
True/False: Encoder-only models typically use causal masking like GPT. Answer: False—they use bidirectional self-attention.
Multiple Choice: Cross-attention appears in: (a) pure encoder-only BERT, (b) encoder–decoder Transformers, (c) never. Answer: (b).
Short Answer: Which Module 10.1 lecture covers QKV projections for multiple heads? Answer: Multi-Head Attention.
Short Answer: Why is LayerNorm used in Transformer blocks? Answer: To stabilize activations/gradients across depth.
Multiple Choice: Decoder-only LMs primarily use: (a) bidirectional MLM, (b) causal self-attention, (c) only pooling. Answer: (b).
True/False: A Transformer layer preserves sequence length when there is no pooling/striding. Answer: True (shapes stay B×L×d_model).

Key Takeaways

A Transformer is a repeated attention + FFN block with residuals and LayerNorm.
Layouts specialize: encoder–decoder, encoder-only, decoder-only.
Capacity knobs: d_model, heads, FFN width, depth.
Module 10.1 primitives implement the contract of each layer.
Next: Transformer Architecture (Full Stack).

Trainer’s Guide

Hands-on idea: Have students modify nhead and print parameter counts; discuss what capacity each knob buys.

Discussion prompt: When would you choose encoder-only vs. decoder-only for a new product feature?

Recap: The Transformer is a deep stack of attention–FFN blocks. Continue with the full stack.