Encoder Block — Module 10.2

How This Lesson Fits the Module & Volume

The full stack showed where the encoder sits. This lecture opens one encoder block: multi-head self-attention, position-wise FFN, residuals, and LayerNorm—the repeating unit of BERT-style models and the encoder half of NMT Transformers.

It reuses Module 10.1’s self-attention, MHA, FFN, LayerNorm, and residuals, and pairs with the upcoming decoder block.

Learning Objectives

By the end of this lesson, students should be able to:

Draw the encoder block: MHSA → Add&Norm → FFN → Add&Norm (or pre-norm order).
Explain why encoder self-attention is bidirectional (no causal mask).
Contrast post-norm (original paper) vs pre-norm (modern practice).
Implement an encoder block in PyTorch and verify shape preservation.
State how padding masks interact with encoder attention.
Relate stacked encoder blocks to contextual representations for downstream heads.

Definition

An encoder block (encoder layer) is one Transformer layer that applies multi-head self-attention over the input sequence, then a position-wise feed-forward network, each wrapped with residual connections and LayerNorm. Stacking N identical blocks yields the Transformer encoder.

Inside One Block

MHSA

Tokens attend to all tokens.

Residual + Norm

Identity path + LayerNorm.

FFN

Per-position MLP.

Residual + Norm

Stabilize before next layer.

Sublayer	Role	10.1 / 10.2 link
Multi-head self-attention	Contextual mixing	MHSA
Residual / skip	Gradient & identity path	Skip, Residual
LayerNorm	Activation stability	LayerNorm
FFN	Nonlinear feature transform	FFN

Pre-Norm vs Post-Norm

Post-norm (original)

x = Norm(x + Sublayer(x))
Used in the 2017 paper.
Can be trickier to train very deep.

Pre-norm (modern)

x = x + Sublayer(Norm(x))
Common in GPT-style stacks.
Often more stable at depth.

Shared idea

Same MHSA + FFN content.
Only Norm placement differs.
Pick one and stay consistent.

Code: Encoder Block (Pre-Norm)

import torch
from torch import nn

class EncoderBlock(nn.Module):
    def __init__(self, d_model=64, nhead=4, dim_ff=256, dropout=0.1):
        super().__init__()
        self.attn = nn.MultiheadAttention(d_model, nhead, dropout=dropout, batch_first=True)
        self.ff = nn.Sequential(
            nn.Linear(d_model, dim_ff),
            nn.GELU(),
            nn.Dropout(dropout),
            nn.Linear(dim_ff, d_model),
            nn.Dropout(dropout),
        )
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)

    def forward(self, x, key_padding_mask=None):
        # x: (B, L, d_model); key_padding_mask: (B, L) True = PAD
        h = self.norm1(x)
        attn_out, _ = self.attn(h, h, h, key_padding_mask=key_padding_mask)
        x = x + attn_out
        x = x + self.ff(self.norm2(x))
        return x

block = EncoderBlock()
x = torch.randn(2, 12, 64)
pad = torch.zeros(2, 12, dtype=torch.bool)
pad[:, -2:] = True  # last two tokens are padding
print(block(x, key_padding_mask=pad).shape)  # torch.Size([2, 12, 64])

Strengths and Tradeoffs

Strengths

Bidirectional context—ideal for understanding tasks.
Identical blocks stack cleanly to arbitrary depth.
Padding masks cleanly ignore PAD positions.

Tradeoffs

Cannot generate left-to-right without a decoder/causal mask.
Full attention is O(L²) per layer.
Needs enough data to learn rich bidirectional features.

Common Misconception

“Encoder attention is the same as decoder attention.” Encoder MHSA is bidirectional: every position may attend to every other (except pads). Decoder self-attention is typically causally masked so positions cannot see the future.

Knowledge Check

Short Answer: What are the two main sublayers of an encoder block? Answer: Multi-head self-attention and the position-wise FFN.
True/False: Encoder self-attention uses a causal mask by default. Answer: False—it is bidirectional.
Multiple Choice: Pre-norm applies LayerNorm: (a) after the residual add, (b) before the sublayer, (c) only at the output head. Answer: (b).
Short Answer: What does a padding mask prevent? Answer: Attention mass on PAD positions.
True/False: An encoder block changes sequence length. Answer: False—it preserves (B, L, d_model).
Multiple Choice: The FFN operates: (a) across all positions jointly via convolution, (b) independently per position, (c) only on [CLS]. Answer: (b).
Short Answer: Which Module 10.1 lecture explains Q, K, V inside attention? Answer: Query / Key / Value (or Attention).
Short Answer: Name one model family built from stacked encoder blocks. Answer: BERT (or RoBERTa, etc.).
Multiple Choice: Residuals in the encoder primarily: (a) reduce vocab size, (b) provide an identity path for stable depth, (c) replace LayerNorm. Answer: (b).
True/False: Post-norm and pre-norm change Norm placement, not the core MHSA/FFN roles. Answer: True.

Key Takeaways

Encoder block = MHSA + FFN with residuals and LayerNorm.
Attention is bidirectional; pad masks still apply.
Pre-norm vs post-norm is a training-stability choice.
Stacks of encoder blocks power BERT-like understanding models.
Next: Decoder Block.

Trainer’s Guide

Hands-on idea: Compare attention weights with and without a padding mask; visualize that PAD columns receive near-zero mass when masked.

Discussion prompt: Why stack many identical encoder blocks instead of one very wide block?

Recap: The encoder block is the bidirectional Transformer layer. Continue with the Decoder Block.