Batch Normalization — Module 6.1

How This Lesson Fits the Module

Deep networks suffer from internal covariate shift—each layer’s input distribution drifts as weights update. Batch Normalization (BatchNorm) normalizes activations per batch, stabilizing training and allowing higher learning rates.

BatchNorm enabled very deep CNNs before Residual Networks made depth routine. It complements dropout (often less dropout when BN is used).

Learning Objectives

By the end of this lesson, students should be able to:

State BatchNorm: normalize, scale, and shift with learnable γ, β.
Place nn.BatchNorm1d / BatchNorm2d correctly (after linear/conv, before activation is common).
Explain train (batch stats) vs eval (running mean/var).
Relate batch size to BatchNorm stability.
Build a BN + ReLU block for an MLP or small CNN.

BatchNorm Mechanics

For activations x over a mini-batch, compute mean μ and variance σ², then:

x̂ = (x − μ) / √(σ² + ε), output y = γx̂ + β

Learnable γ and β let the layer recover identity if normalization hurts.

Definition — Batch Normalization

Batch normalization standardizes layer inputs across the batch dimension (and spatial dims for conv), then applies affine parameters. Running statistics aggregate batch moments for stable eval() inference.

Module	Normalizes Over	Typical Use
`BatchNorm1d`	(N, C) or (N, C, L)	MLP, 1D conv
`BatchNorm2d`	(N, C, H, W) per channel	CNNs
`LayerNorm`	Features per token (preview)	Transformers

PyTorch BatchNorm Block

import torch.nn as nn

def conv_block(in_ch, out_ch):
    return nn.Sequential(
        nn.Conv2d(in_ch, out_ch, kernel_size=3, padding=1, bias=False),
        nn.BatchNorm2d(out_ch),
        nn.ReLU(inplace=True),
    )

class SmallCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.features = nn.Sequential(
            conv_block(1, 32),
            conv_block(32, 64),
            nn.AdaptiveAvgPool2d(1),
        )
        self.classifier = nn.Linear(64, 10)

    def forward(self, x):
        x = self.features(x).flatten(1)
        return self.classifier(x)

model = SmallCNN()
model.train()  # BN uses batch statistics
model.eval()   # BN uses running_mean / running_var

Batch Size Warning BatchNorm with batch size 1–2 produces noisy stats—use larger batches, GroupNorm, or LayerNorm in those regimes.

Benefits

Faster convergence, higher stable lr
Mild regularization effect
Reduces sensitivity to initialization

Caveats

Batch-dependent—awkward for tiny batches
Train/eval mismatch if eval forgotten
Not ideal for variable-length RNNs (historically)

Common Misconception: “BatchNorm and dropout do the same thing.”

Reality: BatchNorm stabilizes scale; dropout randomly removes units. Papers often use both lightly or favor BN in conv stacks.

Common Misconception: “BatchNorm always goes after ReLU.”

Reality: Pre-activation ResNets use BN before activation. Both orders appear in literature—match your architecture family.

Critical Mistake — Eval Mode on Deployment

Serving a model with model.train() makes BatchNorm and Dropout stochastic—predictions become inconsistent run-to-run. Always eval() in production inference.

Knowledge Check

Short Answer: What learnable params does BatchNorm add per channel? Answer: gamma (scale) and beta (shift).
True/False: At eval, BatchNorm uses mini-batch mean only. Answer: False—running statistics.
Multiple Choice: BatchNorm2d normalizes across: (a) batch+spatial per channel, (b) channels only, (c) time only, (d) loss. Answer: (a).
Short Answer: Why set Conv2d bias=False before BN? Answer: BN beta already shifts; redundant bias.
True/False: Small batch sizes hurt BN estimate quality. Answer: True.
Multiple Choice: Enables higher lr often because: (a) stable activations, (b) more dropout, (c) smaller model, (d) no gradients. Answer: (a).
Short Answer: train() vs eval() effect on BN? Answer: Batch stats vs running stats.
True/False: BatchNorm removes need for any regularization. Answer: False.
Multiple Choice: Module 6.1 capstone on depth: (a) Residual Networks, (b) Sigmoid, (c) Batch, (d) Perceptron. Answer: (a).
Short Answer: epsilon in BN denominator prevents? Answer: Division by zero / numerical instability.

Key Takeaways

BatchNorm normalizes activations; γ/β restore expressiveness.
Training uses batch stats; inference uses running averages.
Conv blocks: Conv → BN → ReLU is a classic pattern.
Needs reasonable batch size for stable estimates.
Next: Residual Networks — Module 6.1 capstone.

Trainer’s Guide

Compare: Train small CNN with/without BN for 10 epochs; overlay loss curves.

Production drill: Demonstrate prediction variance when eval() is forgotten.

What’s Next Module 6.1 closes with Residual Networks — then Module 6.2 Training Loop.