Skip Connection — Module 10.2

How This Lesson Fits the Module & Volume

Module 10.1 introduced the residual connection. In Module 10.2 architecture diagrams you will also hear skip connection—the same identity highway around attention and FFN sublayers that makes deep Transformer stacks trainable.

This lecture reinforces the identity path inside encoder and decoder blocks, ties it to LayerNorm placement, and prepares the residual pattern reused in ViT.

Learning Objectives

By the end of this lesson, students should be able to:

Equate skip connections with residual (identity) additions around Transformer sublayers.
Write the update x = x + Sublayer(x) (or pre-norm variant) and explain why it helps depth.
Relate skips to gradient flow and the ability to learn near-identity layers.
Implement residual wrappers around attention and FFN in PyTorch.
Compare terminology: skip vs residual vs highway (high level).
Spot missing residuals as a common bug when hand-rolling blocks.

Definition

A skip connection (residual connection) adds a layer’s input to its transformed output: y = F(x) + x. In Transformers, F is typically multi-head attention or the position-wise FFN (after optional LayerNorm in the pre-norm formulation). The +x path is the identity shortcut.

Why Skips Exist

Deep stack

Many attention/FFN layers.

Identity path

Signal can bypass F.

Easier grads

Direct routes backward.

Stable training

Depth becomes practical.

Term	In this course
Residual connection (10.1)	Same math: y = F(x) + x
Skip connection (10.2)	Architecture synonym emphasizing the bypass arrow
Add & Norm	Skip plus LayerNorm (post-norm packaging)
Pre-norm residual	x = x + F(Norm(x))

Skip vs Related Ideas

Transformer skip

Element-wise add of same-shaped tensors.
Around MHSA and FFN.
Pairs with LayerNorm.

CNN residual (ResNet)

Same identity idea on feature maps.
May use 1×1 proj if shapes differ.
Historical inspiration for deep nets.

Without skips

Deep stacks harder to optimize.
Layers forced to preserve signal alone.
Common student omission when coding.

Code: Residual Around Attention & FFN

import torch
from torch import nn

class ResidualAdd(nn.Module):
    """Skip connection: y = x + dropout(sublayer(norm(x)))  [pre-norm]."""
    def __init__(self, dim, sublayer, dropout=0.1):
        super().__init__()
        self.norm = nn.LayerNorm(dim)
        self.sublayer = sublayer
        self.drop = nn.Dropout(dropout)

    def forward(self, x, **kwargs):
        return x + self.drop(self.sublayer(self.norm(x), **kwargs))

class TinyBlock(nn.Module):
    def __init__(self, d_model=64, nhead=4):
        super().__init__()
        attn = nn.MultiheadAttention(d_model, nhead, batch_first=True)
        self.res_attn = ResidualAdd(
            d_model,
            lambda h, **kw: attn(h, h, h, need_weights=False, **kw)[0],
        )
        ff = nn.Sequential(nn.Linear(d_model, 4*d_model), nn.GELU(), nn.Linear(4*d_model, d_model))
        self.res_ff = ResidualAdd(d_model, lambda h, **kw: ff(h))

    def forward(self, x):
        x = self.res_attn(x)
        x = self.res_ff(x)
        return x

block = TinyBlock()
x = torch.randn(2, 16, 64)
y = block(x)
print(y.shape)                      # torch.Size([2, 16, 64])
print((y - x).abs().mean().item())  # nonzero: sublayers changed x, but identity was available

Strengths and Tradeoffs

Strengths

Enables deep Transformer training.
Lets layers learn incremental refinements.
Simple to implement—one addition.

Tradeoffs

Requires matching shapes (no silent broadcast bugs).
Still needs LayerNorm / careful init for stability.
Does not by itself fix bad masking or LR choices.

Common Misconception

“Skip connection means the sublayer is skipped during training.” The sublayer still runs; the skip adds its input alongside the sublayer output. At initialization, F can be near zero so the block behaves like identity—then gradually learns useful residuals.

Knowledge Check

Short Answer: Write the residual/skip formula. Answer: y = F(x) + x (or x = x + F(Norm(x)) in pre-norm).
True/False: In this course, skip connection and residual connection refer to the same core idea. Answer: True.
Multiple Choice: Skips primarily help: (a) vocabulary size, (b) training deep stacks via identity paths, (c) tokenization. Answer: (b).
Short Answer: Around which two Transformer sublayers do skips appear? Answer: Multi-head attention and the FFN.
True/False: A skip connection removes the need for LayerNorm. Answer: False—they are complementary.
Multiple Choice: Pre-norm residual is: (a) Norm(x + F(x)), (b) x + F(Norm(x)), (c) F(x) only. Answer: (b).
Short Answer: Which Module 10.1 lecture introduced residuals? Answer: Residual Connection.
Short Answer: What must match for x + F(x) to work? Answer: The tensor shapes of x and F(x).
Multiple Choice: If you forget the +x in a block, you: (a) improve BLEU automatically, (b) often hurt trainability, (c) change the tokenizer. Answer: (b).
True/False: ViT encoder blocks also use residual/skip connections. Answer: True.

Key Takeaways

Skip = residual identity path: y = F(x) + x.
Essential around MHSA and FFN in every Transformer block.
Same idea as Module 10.1 residual connection; 10.2 stresses the architecture diagram.
Pairs with LayerNorm (pre- or post-norm).
Next (capstone): Vision Transformer.

Trainer’s Guide

Hands-on idea: Abate residuals in TinyBlock (return only F(x)) and compare training curves on a tiny copy task.

Discussion prompt: Why is an identity shortcut more powerful than hoping each layer learns to copy its input unaided?

Recap: Skip connections are the residual highways that make deep Transformers trainable. Cap the volume with the Vision Transformer.