LeNet proved CNNs work, but stayed confined to tiny grayscale digits. AlexNet (Krizhevsky, Sutskever & Hinton, 2012) took the same conv–pool–dense template and scaled it to 1.2 million color images across 1,000 classes—then won the ImageNet challenge by a landslide. This single result launched the modern deep learning era.
Learning Objectives
By the end of this lesson, students should be able to:
- Explain what AlexNet changed relative to LeNet and why it mattered.
- Identify the three key innovations: ReLU, dropout, and GPU training.
- Describe the 8-layer AlexNet architecture (5 conv + 3 FC).
- Load a pretrained AlexNet from
torchvision.models. - Explain the role of the 2012 ImageNet victory in deep learning history.
- Recognize AlexNet’s limitations that motivated VGG and ResNet.
AlexNet is an 8-layer deep CNN (5 convolutional + 3 fully connected) that won the 2012 ImageNet Large Scale Visual Recognition Challenge (ILSVRC), cutting top-5 error from ~26% to ~15%. It popularized ReLU activations, dropout regularization, data augmentation, and GPU-accelerated training.
What Changed Since LeNet
| Aspect | LeNet (1998) | AlexNet (2012) |
|---|---|---|
| Activation | tanh (saturating) | ReLU (non-saturating, fast) |
| Parameters | ~60K | ~60M |
| Layers | 7 | 8 (5 conv + 3 FC) |
| Regularization | None explicit | Dropout + data augmentation |
| Hardware | CPU | 2 × GTX 580 GPUs |
| Data | MNIST (60K digits) | ImageNet (1.2M images, 1000 classes) |
| Pooling | Average | Overlapping max pooling |
The Three Breakthroughs
1. ReLU. Replacing tanh with ReLU avoided saturating gradients, training several times faster and enabling deeper networks. 2. Dropout. Randomly zeroing FC-layer activations during training curbed overfitting in the 60M-parameter dense layers. 3. GPUs. Splitting the network across two GPUs made training on 1.2M images feasible in days rather than months.
Architecture
| Layer | Type | Detail |
|---|---|---|
| Conv1 | Conv 11×11, stride 4, 96 filters | Large receptive field, aggressive downsample |
| Conv2 | Conv 5×5, 256 filters | Max pool between |
| Conv3–5 | Conv 3×3, 384/384/256 | Stacked without pooling between 3–4 |
| FC6 | Fully connected, 4096 | Dropout 0.5 |
| FC7 | Fully connected, 4096 | Dropout 0.5 |
| FC8 | Fully connected, 1000 | Softmax over ImageNet classes |
Loading AlexNet in PyTorch
You rarely build AlexNet from scratch today; torchvision ships pretrained weights. This is your first taste of transfer learning.
features backbone reuses learned filters—the essence of transfer learning. Dropout comes straight from Module 6.1.Reality: LeNet predates it by 14 years. AlexNet’s contribution was scaling CNNs—proving that with enough data, compute, ReLU, and regularization, the LeNet template could dominate large-scale vision.
Pretrained AlexNet expects inputs resized to 224×224 and normalized with ImageNet mean/std. Feeding raw pixels or the wrong normalization silently wrecks accuracy. Always use weights.transforms() so preprocessing matches training.
Knowledge Check
- Short Answer: What competition did AlexNet win, and in what year? Answer: ImageNet (ILSVRC), 2012.
- True/False: AlexNet introduced ReLU as its activation. Answer: True (popularized it for deep CNNs).
- Multiple Choice: AlexNet has how many weight layers? (a) 5, (b) 7, (c) 8, (d) 16. Answer: (c) — 5 conv + 3 FC.
- Short Answer: Which regularization technique reduced overfitting in the dense layers? Answer: Dropout.
- True/False: AlexNet was trained on a single CPU. Answer: False—two GPUs.
- Multiple Choice: Roughly how many parameters? (a) 60K, (b) 6M, (c) 60M, (d) 600M. Answer: (c).
- Short Answer: Name one advantage of ReLU over tanh for deep nets. Answer: Non-saturating gradients / faster training.
- True/False: AlexNet’s first conv layer uses a small 3×3 kernel. Answer: False—11×11 with stride 4.
- Multiple Choice: Reusing AlexNet’s conv layers on a new task is called: (a) dropout, (b) transfer learning, (c) pooling, (d) augmentation. Answer: (b).
- Short Answer: Which later model pushed uniform 3×3 stacked depth? Answer: VGG.
Key Takeaways
- AlexNet (2012) scaled the LeNet template to ImageNet and won ILSVRC decisively.
- Its breakthroughs: ReLU, dropout, data augmentation, and GPU training.
- 8 layers, ~60M parameters—most in the huge fully connected layers.
- It launched the deep-learning-for-vision era and made transfer learning practical.
- Next: VGG16 shows that stacking small 3×3 convs beats large kernels.
Demo: Run pretrained AlexNet on a few real photos and print the top-5 predicted classes—students see a 2012 model still recognizing everyday objects.
Discussion: Where do AlexNet’s 60M parameters live? (Answer: the FC6/FC7 4096-wide layers.) This motivates why later networks shrink or remove giant FC layers.