Decoder Block — Module 10.2

How This Lesson Fits the Module & Volume

The encoder block builds bidirectional context. The decoder block generates: it uses causal self-attention, optional cross-attention into encoder memory, then an FFN—again with residuals and LayerNorm.

This is the heart of original NMT Transformers and (without cross-attention) of GPT-style LMs that lead into Vol. 11 Language Models. Masking details continue in Masked Attention.

Learning Objectives

By the end of this lesson, students should be able to:

List the three sublayers of a classic decoder block (masked MHSA, cross-attn, FFN).
Explain why causal masking is required for autoregressive training.
Describe how cross-attention uses encoder memory as K/V.
Contrast encoder–decoder vs decoder-only (GPT) blocks.
Implement a decoder block sketch in PyTorch with attn_mask.
Connect decoder stacks to next-token prediction in Vol. 11.

Definition

A decoder block is a Transformer layer that (1) applies masked multi-head self-attention so each position attends only to past/current tokens, (2) optionally applies multi-head cross-attention to encoder outputs, and (3) applies a position-wise FFN—each with residual connections and LayerNorm.

Three Sublayers

Masked MHSA

Causal self-attention.

Cross-Attn

Q from decoder; K/V from encoder.

FFN

Per-position transform.

Norms + Skips

Stabilize the stack.

Sublayer	Q source	K/V source
Masked self-attention	Decoder states	Decoder states (causal)
Cross-attention	Decoder states	Encoder memory
FFN	Per-position MLP (no QKV)

Encoder–Decoder vs Decoder-Only

Classic decoder

Masked MHSA + cross-attn + FFN.
Needs an encoder stack.
MT, summarization (T5/BART-style).

Decoder-only (GPT)

Masked MHSA + FFN only.
No cross-attention module.
Prefix as “context” in the same stream.

Shared rules

Causal mask for autoregression.
Residuals + LayerNorm.
Same QKV attention math from 10.1.

Code: Decoder Block with Causal Mask

import torch
from torch import nn

def causal_mask(T, device=None):
    # True means "ignore" for nn.MultiheadAttention attn_mask when float?-use float -inf
    m = torch.triu(torch.ones(T, T, device=device), diagonal=1)
    return m.masked_fill(m == 1, float("-inf"))

class DecoderBlock(nn.Module):
    def __init__(self, d_model=64, nhead=4, dim_ff=256, dropout=0.1, use_cross=True):
        super().__init__()
        self.use_cross = use_cross
        self.self_attn = nn.MultiheadAttention(d_model, nhead, dropout=dropout, batch_first=True)
        if use_cross:
            self.cross_attn = nn.MultiheadAttention(d_model, nhead, dropout=dropout, batch_first=True)
        self.ff = nn.Sequential(
            nn.Linear(d_model, dim_ff), nn.GELU(), nn.Linear(dim_ff, d_model)
        )
        n = 3 if use_cross else 2
        self.norms = nn.ModuleList([nn.LayerNorm(d_model) for _ in range(n)])

    def forward(self, x, memory=None):
        B, T, _ = x.shape
        h = self.norms[0](x)
        attn_out, _ = self.self_attn(h, h, h, attn_mask=causal_mask(T, x.device))
        x = x + attn_out
        if self.use_cross:
            assert memory is not None
            h = self.norms[1](x)
            cross_out, _ = self.cross_attn(h, memory, memory)
            x = x + cross_out
            x = x + self.ff(self.norms[2](x))
        else:
            x = x + self.ff(self.norms[1](x))
        return x

dec = DecoderBlock(use_cross=True)
tgt = torch.randn(2, 7, 64)
mem = torch.randn(2, 10, 64)
print(dec(tgt, mem).shape)  # torch.Size([2, 7, 64])

Strengths and Tradeoffs

Strengths

Native support for left-to-right generation.
Cross-attention cleanly conditions on source memory.
Decoder-only simplification scales extremely well for LMs.

Tradeoffs

Causal mask blocks future context during encoding of the target.
Autoregressive inference is sequential (mitigated by KV cache in Vol. 11).
Full encoder–decoder is more modules to train and serve.

Common Misconception

“Teacher forcing means the model can see future target tokens.” During training we feed gold previous tokens, but the causal mask still blocks attending to future positions. Without that mask, the model would cheat by reading the answer it is supposed to predict.

Knowledge Check

Short Answer: Name the three classic decoder sublayers. Answer: Masked self-attention, cross-attention, and FFN.
True/False: Causal masking prevents attending to future positions. Answer: True.
Multiple Choice: In cross-attention, keys and values come from: (a) the decoder only, (b) encoder memory, (c) the vocabulary. Answer: (b).
Short Answer: How does a GPT block differ from a classic NMT decoder block? Answer: It drops cross-attention (masked MHSA + FFN only).
True/False: Decoder blocks still use residual connections and LayerNorm. Answer: True.
Multiple Choice: Teacher forcing without a causal mask would: (a) be fine, (b) allow cheating via future tokens, (c) remove embeddings. Answer: (b).
Short Answer: Which Module 10.1 lecture covers attending from decoder to encoder? Answer: Cross-Attention.
Short Answer: Why is decoder inference sequential? Answer: Each new token depends on previously generated tokens under the causal factorization.
Multiple Choice: Masked MHSA Q/K/V all come from: (a) encoder, (b) decoder states, (c) random noise. Answer: (b).
True/False: Vol. 11 language models build primarily on decoder-style stacks. Answer: True.

Key Takeaways

Decoder block = masked MHSA (+ optional cross-attn) + FFN with norms/skips.
Causal masks enforce autoregressive information flow.
Cross-attention links decoder queries to encoder memory K/V.
GPT-style models are decoder-only cousins of this block.
Next: Multi Head Self Attention.

Trainer’s Guide

Hands-on idea: Print the causal mask matrix for T=5 and have students verify upper-triangle −∞ entries.

Discussion prompt: When is explicit cross-attention better than stuffing source and target into one decoder-only stream?

Recap: Decoder blocks generate under causal constraints and may read encoder memory via cross-attention. Continue with Multi Head Self Attention.