Dropout — Module 6.1

How This Lesson Fits the Module

Large networks memorize training data—the overfitting problem from Volume 05 hits harder with millions of parameters. Dropout randomly disables neurons during training, forcing redundant representations.

Pair dropout with tuned learning rate and weight decay. It differs from batch normalization, which stabilizes activations—both appear in modern MLPs and CNNs.

Learning Objectives

By the end of this lesson, students should be able to:

Explain dropout as training-time stochastic regularization.
Place nn.Dropout(p) after activations in MLPs correctly.
Distinguish train vs eval behavior (dropout off at inference).
Choose dropout rates (typical 0.1–0.5) for hidden layers.
Avoid dropout on output layers where it breaks calibration.

How Dropout Works

Each forward pass, each neuron survives with probability 1 − p. Surviving activations are scaled by 1/(1−p) so expected magnitude matches inference (inverted dropout).

Definition — Dropout

Dropout is a regularization technique that zeroes random activations during training. It approximates training an ensemble of thinned networks and averaging them—without the inference cost of literal ensembling.

Mode	Dropout Behavior	PyTorch Call
Training	Random mask, scale survivors	`model.train()`
Inference	Identity (all neurons active)	`model.eval()`
MC Dropout	Dropout on at test for uncertainty	Special case, research tool

PyTorch Dropout Layer

import torch.nn as nn

class MLPWithDropout(nn.Module):
    def __init__(self, d_in=784, d_hid=256, n_classes=10, p=0.3):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(d_in, d_hid),
            nn.ReLU(),
            nn.Dropout(p),
            nn.Linear(d_hid, d_hid),
            nn.ReLU(),
            nn.Dropout(p),
            nn.Linear(d_hid, n_classes),
        )

    def forward(self, x):
        return self.net(x.view(x.size(0), -1))

model = MLPWithDropout()
model.train()   # dropout active
model.eval()    # dropout disabled for validation

Placement Rule Dropout after activation, before the next linear layer. Do not stack dropout on the final logits layer for standard classification.

When Dropout Helps

Large fully connected layers
Small/medium datasets vs model capacity
Co-adapting feature detectors in deep MLPs

When to Reduce / Skip

Modern CNNs often use BN instead/in addition
Very small data + heavy dropout can underfit
Transformer attention has its own regularization

Common Misconception: “Dropout at inference improves accuracy.”

Reality: Standard inference disables dropout. Leaving it on randomizes outputs unless you intentionally use Monte Carlo dropout for uncertainty.

Common Misconception: “p=0.5 is always optimal.”

Reality: 0.2–0.3 is common in FC layers; input layers often use less. Tune on validation loss.

Knowledge Check

Short Answer: What does dropout probability p mean? Answer: Fraction of activations zeroed each forward pass (training).
True/False: Dropout runs during model.eval(). Answer: False—in standard nn.Dropout.
Multiple Choice: Dropout primarily fights: (a) underfitting, (b) overfitting, (c) slow GPU, (d) bad labels. Answer: (b).
Short Answer: Why scale by 1/(1-p)? Answer: Keep expected activation magnitude consistent train vs eval.
True/False: Dropout on output logits is standard practice. Answer: False.
Multiple Choice: Activate dropout with: (a) model.train(), (b) torch.no_grad(), (c) zero_grad, (d) eval(). Answer: (a).
Short Answer: Ensemble interpretation of dropout? Answer: Training many thinned sub-networks randomly.
True/False: Dropout and weight decay address the same issue differently. Answer: True—both regularize.
Multiple Choice: Next stabilization technique: (a) Batch Normalization, (b) Epoch, (c) Softmax, (d) Weights. Answer: (a).
Short Answer: Typical hidden dropout range? Answer: About 0.1 to 0.5 (often 0.2–0.3).

Key Takeaways

Dropout randomly drops activations during training only.
model.train() vs eval() controls dropout and BN.
Use after ReLU in hidden layers; skip final logits.
Regularizes co-adaptation; complements weight decay.
Next: Batch Normalization — stabilizing layer inputs.

Trainer’s Guide

A/B train: Same MLP on MNIST with p=0 vs p=0.3; compare train–val accuracy gap.

Bug hunt: Student forgets model.eval() before test—show noisy predictions.

What’s Next Stabilize deep training with Batch Normalization.