Learning Rate — Module 6.1

How This Lesson Fits the Module

SGD, Adam, and every optimizer share one critical knob: learning rate (η). It controls how far weights move along the gradient each step.

Too high → divergence or oscillation. Too low → glacial progress. Schedules tie learning rate to epochs and batch size.

Learning Objectives

By the end of this lesson, students should be able to:

Interpret learning rate as step size in parameter space.
Diagnose too-high vs too-low learning rate from loss curves.
Use PyTorch learning rate schedulers (StepLR, CosineAnnealingLR).
Run a simple learning rate range test before full training.
Coordinate LR changes with batch size scaling rules.

Step Size Intuition

Gradient points downhill. Learning rate scales that direction: Δθ = −η∇L. In a valley, large η bounces between walls; tiny η creeps slowly.

Definition — Learning Rate

The learning rate is a positive scalar (or per-group hyperparameter) multiplying gradients before weight updates. It is often the first hyperparameter tuned after fixing architecture and batch size.

Symptom	Likely LR Issue	Fix
Loss NaN / explodes	Too high	Reduce lr 10×; check normalization
Loss flat from start	Too low or frozen graph	Increase lr; verify backward
Loss oscillates, no trend	Slightly too high	Lower lr or add warmup
Train ok, val poor	Not LR alone	See Dropout, data issues

Schedulers in PyTorch

import torch.optim as optim
from torch.optim.lr_scheduler import CosineAnnealingLR, SequentialLR, LinearLR

optimizer = optim.AdamW(model.parameters(), lr=1e-3)

warmup = LinearLR(optimizer, start_factor=0.1, total_iters=5)
cosine = CosineAnnealingLR(optimizer, T_max=45)
scheduler = SequentialLR(optimizer, schedulers=[warmup, cosine], milestones=[5])

for epoch in range(50):
    train_one_epoch(model, train_loader, optimizer)
    scheduler.step()
    print(epoch, optimizer.param_groups[0]["lr"])

LR Range Test (Sketch)

Increase lr exponentially over a few hundred batches; plot loss vs lr. Choose lr just before loss spikes—classic diagnostic from Leslie Smith’s LR finder idea.

Common Misconception: “Set lr once and forget.”

Reality: Decaying lr late in training fine-tunes weights in a flatter region. Cosine and step decay are standard for a reason.

Common Misconception: “Same lr works for Adam and SGD.”

Reality: Adam often uses 1e-3 to 3e-4; SGD on ImageNet may use 0.1 with momentum at batch 256. Compare optimizers separately.

Critical Mistake — Scheduler Step Timing

Some schedulers step per epoch (scheduler.step() after epoch loop); others per batch (OneCycleLR). Wrong placement silently uses wrong lr schedule.

Knowledge Check

Short Answer: Role of learning rate in SGD? Answer: Scales the gradient step size.
True/False: Loss NaN often indicates lr too high. Answer: True.
Multiple Choice: CosineAnnealingLR typically: (a) increases lr forever, (b) decays lr smoothly, (c) disables Adam, (d) doubles batch size. Answer: (b).
Short Answer: Why warmup? Answer: Stabilize early updates when moments or large grads are noisy.
True/False: param_groups[0]["lr"] shows current lr. Answer: True.
Multiple Choice: Flat loss from epoch 1 suggests: (a) lr too low or broken grads, (b) perfect model, (c) too much dropout only, (d) eval mode. Answer: (a).
Short Answer: Linear batch-lr scaling: double batch, do what to lr? Answer: Try doubling lr (heuristic).
True/False: Learning rate is the only hyperparameter that matters. Answer: False.
Multiple Choice: Next regularization lecture: (a) Dropout, (b) Forward Propagation, (c) Batch, (d) Perceptron. Answer: (a).
Short Answer: Default Adam lr in PyTorch? Answer: 1e-3.

Key Takeaways

Learning rate is step size—most impactful optimizer hyperparameter.
Watch loss curves for diverge (high) vs stall (low).
Schedulers decay lr over epochs for finer late training.
Warmup helps large models and large effective batches.
Next: Dropout — regularization beyond lr tuning.

Trainer’s Guide

Lab: Train same model with lr 1e-2, 1e-3, 1e-4; students classify curves into healthy / diverged / stalled.

Code review: Find scheduler.step() in wrong loop position in a buggy snippet.

What’s Next Control overfitting with Dropout, not lr alone.