Decoder — Module 10.1

How This Lesson Fits the Module & Volume

The previous lecture introduced the encoder: it reads the source and writes a contextual memory. Generation tasks—translation, summarization, dialogue—need a second half that produces the target sequence one token at a time while consulting that memory.

This lecture defines the decoder. In Module 10.1 you will soon learn the Q/K/V vocabulary and see how cross-attention lets the decoder query encoder states. For now, focus on the decoder’s dual duties: causal self-modeling of the target so far, and reading the encoder.

Learning Objectives

By the end of this lesson, students should be able to:

Define a decoder as an autoregressive generator conditioned on encoder memory.
Explain causal (masked) self-attention and why it prevents future leakage.
Distinguish decoder self-attention from cross-attention to the encoder.
Describe teacher forcing during training versus greedy/beam search at inference.
Sketch a Transformer decoder block and implement a minimal PyTorch decoder path.
Relate encoder–decoder roles to the Query/Key/Value lectures that follow.

Definition

A decoder is a neural module that generates an output sequence y₁, …, y_m autoregressively. At step t it predicts y_t from previous targets y_<t and—in encoder–decoder models—from encoder memory h_1..n.

Two Attention Paths Inside the Decoder

Masked Self-Attention

Target tokens attend only to past (and self).
Enforces left-to-right generation.
Prevents peeking at future labels.

Cross-Attention

Queries from decoder; keys/values from encoder.
Aligns each output step to source tokens.
Detailed in Cross Attention.

Feed-Forward

Same position-wise MLP pattern as the encoder.
Plus residuals and LayerNorm.
Stacked N times.

Encoder vs. Decoder at a Glance

Aspect	Encoder	Decoder
Input	Source tokens	Target tokens (shifted)
Self-attention	Full (bidirectional)	Causal (masked)
Cross-attention	None	To encoder memory
Typical goal	Represent	Generate / predict next token
At inference	One forward pass	Loop over steps

Training vs. Inference

During training, teacher forcing feeds ground-truth previous tokens so the model can compute all target positions in parallel (still with a causal mask). At inference, the model must feed its own predictions back in—greedy argmax, sampling, or beam search—so latency grows with output length.

Minimal Decoder Usage in PyTorch

nn.Transformer packages encoder and decoder. The shapes below show the memory bridge from encoder to decoder.

import torch
from torch import nn

d_model, nhead, vocab = 64, 4, 1000
model = nn.Transformer(
    d_model=d_model, nhead=nhead,
    num_encoder_layers=2, num_decoder_layers=2,
    dim_feedforward=128, batch_first=True,
)
src_emb = nn.Embedding(vocab, d_model)
tgt_emb = nn.Embedding(vocab, d_model)

src = torch.randint(0, vocab, (2, 10))   # source length 10
tgt = torch.randint(0, vocab, (2, 8))    # target length 8 (teacher-forced)

# Causal mask: position i cannot see j > i
tgt_mask = nn.Transformer.generate_square_subsequent_mask(8)

out = model(src_emb(src), tgt_emb(tgt), tgt_mask=tgt_mask)
print(out.shape)   # torch.Size([2, 8, 64]) — per target position

Where Query, Key, and Value Enter

In cross-attention, decoder states become queries (“what am I looking for?”), while encoder states provide keys and values (“what is available to read?”). That vocabulary is the next three lectures.

Strengths

Flexible generation conditioned on rich memory.
Causal mask matches real left-to-right decoding.
Cross-attention soft-aligns to any source position.

Tradeoffs

Autoregressive inference is sequential and slower.
Exposure bias: train on gold prefixes, test on own errors.
Must carefully mask PAD and future tokens.

Common Misconception

“Decoder self-attention is the same as encoder self-attention.” Encoder self-attention is bidirectional over the source. Decoder self-attention is causal: each position may attend only to earlier target positions. Without that mask, the model would cheat by reading future answers during training.

Knowledge Check

Short Answer: What two information sources does a seq2seq decoder use at step t? Answer: Previous target tokens y<t and encoder memory h1..n.
True/False: Decoder self-attention is typically bidirectional over the full target. Answer: False—it is causally masked.
Multiple Choice: Cross-attention pulls keys/values from: (a) the decoder only, (b) the encoder memory, (c) the optimizer. Answer: (b).
Short Answer: What is teacher forcing? Answer: Feeding ground-truth previous tokens during training instead of the model’s own predictions.
True/False: At inference, the decoder usually runs in a loop, one new token per step. Answer: True.
Multiple Choice: A causal mask stops a position from attending to: (a) PAD only, (b) future positions, (c) the encoder. Answer: (b).
Short Answer: Name one inference decoding strategy. Answer: Greedy argmax, sampling, or beam search (any one).
True/False: The encoder and decoder always share the same embedding table. Answer: False—they often use separate source/target embeddings.
Multiple Choice: In the PyTorch example, out.shape matches: (a) source length, (b) target length, (c) vocab size only. Answer: (b).
Short Answer: In cross-attention, which side provides the queries? Answer: The decoder.

Key Takeaways

Decoders generate targets autoregressively, conditioned on encoder memory.
Causal self-attention + cross-attention + FFN define the Transformer decoder block.
Training uses teacher forcing; inference feeds predictions back step by step.
Next: Query—the “what am I looking for?” vector in attention.

Trainer’s Guide

Hands-on idea: Print tgt_mask and have students verify the upper triangle is masked.

Discussion prompt: Why would removing the causal mask inflate training accuracy but break generation?

Recap: The decoder reads encoder memory and writes the target under a causal constraint. Continue with Query.