Residual Connection — Module 10.1

How This Lesson Fits the Module & Engineering Practice

Deep stacks of multi-head attention and FFNs would be brittle without skip paths. Residual connections add the block input to its transformed output so gradients and identity signals can flow through many layers.

You already met this idea in Vol 06 — Residual Networks and saw it scale vision models in Vol 07 — ResNet. Transformers reuse the same principle around attention and FFN sublayers.

Learning Objectives

By the end of this lesson, students should be able to:

State the residual update \(y = x + F(x)\).
Link transformer residuals to Vol 06 residual nets and Vol 07 ResNet.
Describe residual placement around attention and FFN sublayers.
Implement a residual attention block in PyTorch.
Explain how residuals help train deep transformers.
Avoid dimension mismatches that break skip adds.

Definition

A residual (skip) connection adds a sublayer’s input to its output: \(y = x + F(x)\), where \(F\) is attention, an FFN, or another transform. The network learns a residual refinement instead of a full replacement of \(x\).

From ResNet to Transformers

In vision, residual blocks let CNNs go much deeper by preserving an identity path (see residual networks and ResNet). Transformers apply the same pattern to sequence blocks:

\[x \leftarrow x + \mathrm{MultiHeadAttn}(\ldots)\]

\[x \leftarrow x + \mathrm{FFN}(x)\]

(Normalization placement—pre-norm vs post-norm—varies by design; see Layer Normalization.)

ResNet (Vol 06/07)

Skip over conv blocks.
Enables very deep CNNs.
Often with BatchNorm.

Transformer residual

Skip over attn / FFN.
Enables deep token stacks.
Paired with LayerNorm.

Why Residuals Matter Here

Benefit	Intuition in transformers
Gradient flow	Direct path around nonlinear attn/FFN
Identity bias	Layer can default to “pass through”
Stable depth	Stack many blocks without collapse
Composable features	Each block adds a refinement

1. Input \(x\)

Token states enter the sublayer.

2. Transform \(F(x)\)

Attention or FFN computes an update.

3. Add

Form \(x + F(x)\) (same shape).

4. Normalize

Usually LayerNorm in the block recipe.

PyTorch: Residual Around Attention

import torch
from torch import nn

class ResidualMHA(nn.Module):
    """Post-norm style: x = LayerNorm(x + Dropout(MHA(x)))."""
    def __init__(self, d_model: int, num_heads: int, dropout: float = 0.1):
        super().__init__()
        self.mha = nn.MultiheadAttention(d_model, num_heads, batch_first=True)
        self.dropout = nn.Dropout(dropout)
        self.norm = nn.LayerNorm(d_model)

    def forward(self, x):
        attn_out, _ = self.mha(x, x, x)
        return self.norm(x + self.dropout(attn_out))

x = torch.randn(2, 12, 64)
y = ResidualMHA(64, 8)(x)
print(y.shape)  # torch.Size([2, 12, 64]) — same shape as x

Strengths

Proven depth recipe from ResNet era.
Simple: elementwise add when shapes match.
Works with dropout on the branch.

Requirements

\(F(x)\) must match \(x\)’s last dim.
Needs norm + init for stability.
Wrong pre/post-norm can hurt training.

Common Mistake

Changing feature width inside \(F\) (for example, projecting attention to a different \(d\)) and then trying to add back to \(x\). Residuals require broadcast-compatible shapes—typically identical \((B, T, d_model)\).

Misconception

“Residuals are only for CNNs.” Transformers depend on them just as critically. Without skips, deep attention stacks are much harder to optimize—the same lesson as ResNet, applied to sequences.

Siblings: Self Attention, Positional Encoding, Layer Normalization, Feed-Forward Network, Encoder.

Knowledge Check

Short Answer: Write the residual update formula. Answer: \(y = x + F(x)\).
True/False: Transformer residuals reuse the same idea as ResNet skip connections. Answer: True.
Multiple Choice: Which Vol 07 architecture popularized deep residual CNNs? (a) ResNet, (b) LeNet only, (c) Naive Bayes. Answer: (a).
Short Answer: Name two transformer sublayers wrapped by residuals. Answer: Multi-head attention and the position-wise FFN.
True/False: Residual addition requires matching feature dimensions. Answer: True.
Multiple Choice: A main benefit of residuals is: (a) better gradient/identity flow with depth, (b) removing all norms, (c) eliminating PE. Answer: (a).
Short Answer: Where was residual learning introduced earlier in this curriculum? Answer: Vol 06 Residual Networks (and ResNet in Vol 07).
True/False: Residuals replace the need for LayerNorm. Answer: False; they are usually used together.
Multiple Choice: After residual connection in this module order comes: (a) Layer Normalization, (b) Stemming, (c) Object detection. Answer: (a).
Short Answer: What happens conceptually if \(F(x)\approx 0\)? Answer: The block behaves like an identity map, passing \(x\) through.

Key Takeaways

Transformers use \(x + F(x)\) around attention and FFN, like ResNet skips.
Curriculum anchors: Vol 06 Residual Networks, Vol 07 ResNet.
Shapes must match for the add; \(W_O\) and FFN restore \(d_model\).
Residuals enable deep, trainable attention stacks.
Next: Layer Normalization.

Trainer’s Guide

Hands-on idea: Ablate the skip add in a 6-layer toy transformer and compare training loss curves.

Discussion prompt: How does the residual intuition from ResNet transfer when \(F\) is attention instead of convolution?

Recap: Residual connections let each transformer sublayer learn a refinement \(x + F(x)\), the same depth trick as ResNet. Continue with Layer Normalization.