Batch Size — Module 6.1

How This Lesson Fits the Module

A batch groups examples; batch size is how many. It controls gradient noise, memory use, and how many optimizer steps occur per epoch.

Batch size interacts with learning rate and batch normalization. Treat them as coupled hyperparameters, not isolated knobs.

Learning Objectives

By the end of this lesson, students should be able to:

Relate batch size to GPU memory and steps per epoch.
Describe small-batch vs large-batch training tradeoffs.
Apply the linear LR scaling rule cautiously when increasing batch size.
Choose batch size under memory constraints with gradient accumulation.
Explain batch size effects on BatchNorm statistics.

Tradeoff Landscape

Definition — Batch Size

Batch size (B) is the number of training examples used to compute one gradient estimate before an optimizer step. It is the primary lever for the noise–throughput tradeoff in deep learning.

Batch Size	Gradient	Memory	Typical Effect
Small (8–32)	Noisy	Low	More updates/epoch; may generalize better
Medium (64–256)	Balanced	Moderate	Common default for CNNs on single GPU
Large (512+)	Stable	High	Faster per epoch; may need higher LR, worse sharp minima

Memory and Gradient Accumulation

When a full batch does not fit in VRAM, accumulate gradients over micro-batches, then step once.

accum_steps = 4          # effective batch = micro_batch * accum_steps
optimizer.zero_grad()

for i, (x, y) in enumerate(train_loader):
    loss = criterion(model(x), y) / accum_steps
    loss.backward()
    if (i + 1) % accum_steps == 0:
        optimizer.step()
        optimizer.zero_grad()

Learning Rate Scaling

Heuristic: if you multiply batch size by k, try multiplying learning rate by k (linear scaling rule). Works best for SGD; validate with a short run—Adam often needs less aggressive scaling.

Common Misconception: “Batch size 1 is always best for generalization.”

Reality: Tiny batches are slow on GPUs and can destabilize BatchNorm. Sweet spots are task- and hardware-dependent.

Common Misconception: “Effective batch size only matters for memory.”

Reality: Generalization can differ between one batch of 256 and eight accumulated steps of 32 at the same effective size—though they are often close.

Critical Mistake — OOM Without a Plan

Crashes from CUDA OOM during training waste hours. Start with a small batch, benchmark memory, then scale up or use accumulation instead of blindly setting batch_size=1024.

Knowledge Check

Short Answer: Effect of larger batch on gradient noise? Answer: Lower noise (more stable estimate).
True/False: Doubling batch size always halves steps per epoch. Answer: True (with fixed dataset size).
Multiple Choice: Gradient accumulation simulates: (a) larger batch, (b) more epochs, (c) dropout, (d) weight init. Answer: (a).
Short Answer: Linear LR scaling rule in one sentence? Answer: Scale learning rate proportionally when scaling batch size.
True/False: Large batches always improve validation accuracy. Answer: False.
Multiple Choice: BatchNorm struggles most with: (a) batch size 1, (b) batch size 128, (c) eval mode, (d) ReLU. Answer: (a).
Short Answer: N=5000, B=100, steps per epoch? Answer: 50.
True/False: pin_memory reduces batch size. Answer: False—it speeds host-to-device transfer.
Multiple Choice: Next lecture on full dataset passes: (a) Epoch, (b) Perceptron, (c) ReLU, (d) Weights. Answer: (a).
Short Answer: First thing to try when GPU OOM? Answer: Reduce batch size or use gradient accumulation.

Key Takeaways

Batch size trades gradient noise for memory and throughput.
Steps per epoch = N / B (approximately).
Scale learning rate when changing batch size—then verify.
Gradient accumulation simulates large batches on limited VRAM.
Next: Epoch — counting full passes through the data.

Trainer’s Guide

Benchmark lab: Same model, batch sizes 16/64/256—plot wall time per epoch and validation accuracy.

Pair with BN lecture: Show unstable BN loss with batch_size=2 on a small CNN.

What’s Next Many batches form one epoch.