ResNet — Module 7.3

How This Lesson Fits the Module

VGG showed depth helps—until it does not. Beyond ~20 plain layers, accuracy degrades: gradients vanish and the network cannot even learn the identity function. ResNet (He et al., Microsoft, 2015) fixed this with a deceptively simple idea—the skip connection—and won ILSVRC 2015 with a 152-layer network. It is arguably the most influential vision architecture ever.

VGG (2014) — deep plain stacks, saturates near 19 layers ↓ ResNet (2015) — skip connections; 50/101/152 layers train cleanly ↓ Next: EfficientNet scales depth/width/resolution together

Learning Objectives

By the end of this lesson, students should be able to:

State the residual formulation y = x + F(x) and why it helps optimization.
Explain the degradation problem that motivated ResNet.
Distinguish basic blocks (ResNet-18/34) from bottleneck blocks (ResNet-50+).
Load ResNet-50 from torchvision.models and adapt its head.
Connect ResNet to the residual networks lecture in Vol 06.
Explain why residual connections now appear in almost every deep architecture.

Definition — Residual Block

A residual block computes y = x + F(x), where F is a small stack of conv–BN–ReLU layers and x is added back via a skip (shortcut) connection. F learns only the residual correction; if identity is optimal, F simply learns to output near-zero.

The Degradation Problem

Stacking more plain layers should never hurt—the extra layers could just learn identity. In practice they cannot: optimization struggles to drive a stack of nonlinear layers to identity, so deeper plain nets train worse. Skip connections make identity the default, so depth stops hurting. Gradients also flow directly through the +x path, easing the vanishing-gradient problem (see backpropagation).

Basic vs Bottleneck Blocks

Model	Block type	Layers	Params	Year
ResNet-18	Basic (2 × 3×3)	18	~11.7M	2015
ResNet-34	Basic	34	~21.8M	2015
ResNet-50	Bottleneck (1×1, 3×3, 1×1)	50	~25.6M	2015
ResNet-101	Bottleneck	101	~44.5M	2015
ResNet-152	Bottleneck	152	~60.2M	2015

Note ResNet-50 beats VGG16 in accuracy with ~5× fewer parameters—the bottleneck block (1×1 down, 3×3, 1×1 up) is far more efficient than VGG’s dense stacks.

Residual Block in PyTorch

import torch
import torch.nn as nn
from torchvision.models import resnet50, ResNet50_Weights

# The core idea, hand-written:
class BasicBlock(nn.Module):
    def __init__(self, in_ch, out_ch, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_ch, out_ch, 3, stride, 1, bias=False)
        self.bn1 = nn.BatchNorm2d(out_ch)
        self.conv2 = nn.Conv2d(out_ch, out_ch, 3, 1, 1, bias=False)
        self.bn2 = nn.BatchNorm2d(out_ch)
        self.relu = nn.ReLU(inplace=True)
        self.shortcut = nn.Sequential()
        if stride != 1 or in_ch != out_ch:            # match shapes
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_ch, out_ch, 1, stride, bias=False),
                nn.BatchNorm2d(out_ch))

    def forward(self, x):
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out = out + self.shortcut(x)                  # skip connection
        return self.relu(out)

# In practice, load a pretrained ResNet-50:
model = resnet50(weights=ResNet50_Weights.IMAGENET1K_V2)
model.fc = nn.Linear(model.fc.in_features, 10)        # 10-class head
print(sum(p.numel() for p in model.parameters()))     # ~23.5M after head swap

Vol 06 Tie-In This is the same residual idea developed in Vol 06 — Residual Networks, now realized as a full ImageNet backbone. The BatchNorm and ReLU layers come straight from Module 6.1.

Common Misconception: “Skip connections skip layers, so those layers do nothing.”

Reality: The layers in F still learn—they learn the residual. The skip only guarantees a clean gradient path and an easy identity fallback; it does not disable F.

Critical Mistake — Missing Projection on the Shortcut

When stride or channel count changes, x and F(x) have different shapes and cannot be added. You must apply a 1×1 projection conv on the shortcut (as in the code above). Forgetting it throws a shape-mismatch error at the out + shortcut(x) line.

Knowledge Check

Short Answer: Write the residual block output formula. Answer: y = x + F(x).
True/False: ResNet won ImageNet in 2015. Answer: True.
Multiple Choice: The problem ResNet solved is: (a) overfitting, (b) degradation with depth, (c) slow data loading, (d) class imbalance. Answer: (b).
Short Answer: What kind of block does ResNet-50 use? Answer: Bottleneck (1×1, 3×3, 1×1).
True/False: ResNet-50 has more parameters than VGG16. Answer: False—far fewer (~25M vs ~138M).
Multiple Choice: When channels change, the shortcut needs a: (a) softmax, (b) 1×1 projection conv, (c) dropout, (d) larger LR. Answer: (b).
Short Answer: Why does the skip connection help gradients? Answer: It provides a direct additive path so gradients don’t vanish.
True/False: Residual connections appear only in CNNs. Answer: False—transformers use them too.
Multiple Choice: How many layers is the largest classic ResNet here? (a) 34, (b) 50, (c) 101, (d) 152. Answer: (d).
Short Answer: If identity is the optimal mapping, what should F learn? Answer: A near-zero residual.

Key Takeaways

ResNet introduced skip connections: y = x + F(x).
They cure the degradation problem and let 50–152 layers train cleanly.
Bottleneck blocks make ResNet-50 more accurate and smaller than VGG16.
Residual connections are now standard—including inside transformers.
Next: EfficientNet scales depth, width, and resolution jointly.

Trainer’s Guide

Experiment: Train a 20-layer plain CNN vs a 20-layer residual CNN on CIFAR-10; students watch the plain net’s training loss stall while the residual net keeps improving.

Link forward: Point out that the Vision Transformer also wraps its attention and MLP blocks in residual connections—the idea is universal.

Progress The depth ceiling is broken. Next we ask how to scale a network optimally rather than just deeper. Continue to EfficientNet.