LeNet — Module 7.3

How This Lesson Fits the Module

Module 7.1 taught the building blocks of convolution—kernels, stride, padding, pooling, and feature maps—and Module 7.2 applied them to classification, detection, and segmentation. Module 7.3 now walks the historical arc of the models that made those tasks possible.

LeNet-5 (Yann LeCun, 1998) is where that arc begins. It is the first convolutional neural network deployed at scale—reading handwritten digits on bank checks. Every model in this module is a descendant of the pattern LeNet established: stacked convolutions, downsampling, then fully connected classification.

LeNet-5 (1998) — CNNs work; digits on CPUs ↓ AlexNet (2012) — GPUs + ReLU + ImageNet scale ↓ VGG / ResNet / EfficientNet — depth, residuals, scaling laws

Learning Objectives

By the end of this lesson, students should be able to:

Describe the LeNet-5 architecture layer by layer and its role in CNN history.
Explain the conv → subsample → conv → subsample → dense pattern it introduced.
Reconstruct LeNet-5 in PyTorch and count its parameters.
Relate LeNet components back to the convolution and pooling primitives from Module 7.1.
Explain why LeNet worked on MNIST but could not scale to natural images.
Position LeNet as the direct ancestor of AlexNet.

Why LeNet Matters

Before LeNet, image recognition relied on hand-engineered features fed into a separate classifier. LeNet proved that a single network could learn the features and the classifier jointly through backpropagation. It combined three ideas that still define CNNs today: local receptive fields, shared weights, and spatial subsampling. These give translation tolerance and parameter efficiency that fully connected networks lack.

Definition — LeNet-5

LeNet-5 is a 7-layer convolutional neural network (not counting the input) designed by Yann LeCun et al. for handwritten digit recognition. It alternates convolutional layers (learnable filters) with subsampling / pooling layers, then flattens into fully connected layers ending in a 10-way output for digits 0–9.

Architecture, Layer by Layer

LeNet-5 accepts a 32×32 grayscale image (28×28 MNIST digits padded). It uses tanh activations and average-style subsampling—a product of its 1998 era, before ReLU and max pooling became standard.

Layer	Type	Output shape	Notes
Input	Image	1 × 32 × 32	Grayscale digit
C1	Conv 5×5, 6 filters	6 × 28 × 28	Edge / stroke detectors
S2	Subsample 2×2	6 × 14 × 14	Downsample (avg pool)
C3	Conv 5×5, 16 filters	16 × 10 × 10	Combine strokes
S4	Subsample 2×2	16 × 5 × 5	Downsample
C5	Conv/FC, 120 units	120	Fully connected
F6	Fully connected	84	Dense
Output	Fully connected	10	Digit class scores

Total learnable parameters: roughly 60,000—tiny by modern standards, yet enough to reach ~99% on MNIST. The key: convolution reuses each filter across all spatial positions, so a 5×5 filter with 6 channels needs only 156 parameters instead of one weight per pixel.

LeNet-5 in PyTorch

A faithful reconstruction using modern PyTorch. We keep tanh to honor the original, though ReLU trains faster in practice.

import torch
import torch.nn as nn

class LeNet5(nn.Module):
    def __init__(self, n_classes=10):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 6, kernel_size=5),   # C1: 1x32x32 -> 6x28x28
            nn.Tanh(),
            nn.AvgPool2d(2),                  # S2: -> 6x14x14
            nn.Conv2d(6, 16, kernel_size=5),  # C3: -> 16x10x10
            nn.Tanh(),
            nn.AvgPool2d(2),                  # S4: -> 16x5x5
        )
        self.classifier = nn.Sequential(
            nn.Flatten(),                     # -> 400
            nn.Linear(16 * 5 * 5, 120),       # C5
            nn.Tanh(),
            nn.Linear(120, 84),               # F6
            nn.Tanh(),
            nn.Linear(84, n_classes),         # Output
        )

    def forward(self, x):
        x = self.features(x)
        return self.classifier(x)

model = LeNet5()
n_params = sum(p.numel() for p in model.parameters())
print(f"LeNet-5 parameters: {n_params:,}")   # ~61,706
dummy = torch.randn(1, 1, 32, 32)
print(model(dummy).shape)                     # torch.Size([1, 10])

Module 7.1 Tie-In Each Conv2d here is exactly the convolution operation you studied—a learnable kernel sliding over the input to build a feature map; AvgPool2d is average pooling.

Why It Could Not Scale

LeNet excelled on small, centered, grayscale digits. Natural images (color, cluttered backgrounds, varied lighting, thousands of classes) overwhelmed it. Three limits blocked scaling in 1998: compute (no GPUs), data (no ImageNet), and saturating activations (tanh gradients vanish in deep stacks). AlexNet would remove all three barriers 14 years later.

Common Misconception: “LeNet is obsolete, so there is nothing to learn from it.”

Reality: Every modern CNN—including the backbone of YOLO and Mask R-CNN—still uses LeNet’s core template: stacked convolutions that downsample spatially while growing channel depth, then a classifier head.

Critical Mistake — Forgetting Input Size

The Linear(16 * 5 * 5, 120) layer hard-codes the flattened size. Feed a 28×28 image without padding to 32×32 and the spatial dimensions after S4 change, producing a shape-mismatch error. Always pad MNIST to 32×32 or recompute the flatten dimension.

Knowledge Check

Short Answer: Who created LeNet-5 and in what year? Answer: Yann LeCun et al., 1998.
True/False: LeNet-5 uses ReLU activations. Answer: False—it uses tanh (ReLU came later with AlexNet).
Multiple Choice: LeNet’s core repeating pattern is: (a) attention blocks, (b) conv → subsample, (c) residual add, (d) depthwise conv. Answer: (b).
Short Answer: Roughly how many parameters does LeNet-5 have? Answer: About 60,000.
True/False: Weight sharing in convolution reduces parameters versus a fully connected layer. Answer: True.
Multiple Choice: LeNet was originally deployed to: (a) detect faces, (b) read handwritten digits/checks, (c) caption photos, (d) segment tumors. Answer: (b).
Short Answer: Name one reason LeNet could not scale to natural images in 1998. Answer: Lack of GPU compute / large datasets / saturating tanh gradients (any one).
True/False: The final LeNet layer outputs 10 values for digit classes. Answer: True.
Multiple Choice: The subsampling layers S2 and S4 primarily: (a) add parameters, (b) reduce spatial resolution, (c) increase channels only, (d) apply softmax. Answer: (b).
Short Answer: Which model directly advanced LeNet’s ideas to ImageNet scale? Answer: AlexNet (2012).

Key Takeaways

LeNet-5 (1998) is the first practical CNN and the template for all that follow.
It alternates convolution and subsampling, then classifies with dense layers.
Weight sharing makes it parameter-efficient (~60K params).
Era limits (no GPUs, small data, tanh saturation) capped it to simple digits.
Next: AlexNet scales this template to ImageNet with GPUs and ReLU.

Trainer’s Guide

Lab: Train the LeNet-5 above on MNIST for 5 epochs; students should hit ~98%+ accuracy and observe how quickly a tiny CNN learns.

Discussion: Swap Tanh for ReLU and AvgPool2d for MaxPool2d. Ask students to predict and then measure the change in convergence speed—this previews the AlexNet leap.

Module 7.3 Begins You now have the ancestral CNN. Continue the lineage with AlexNet, the model that ignited the deep learning revolution.