Optimizers — Module 6.1

How This Lesson Fits the Module

Loss functions tell the network how wrong it is. Optimizers decide how to move the weights and biases to reduce that loss. Without an optimizer, backpropagation would compute gradients but never apply them.

This lesson is the map of the optimization landscape. Later lectures drill into SGD, Adam, and how learning rate ties the family together.

Learning Objectives

By the end of this lesson, students should be able to:

Define an optimizer’s role in the training loop: compute gradients, update parameters.
Contrast first-order methods (SGD family) with full-batch gradient descent.
Identify common PyTorch optimizers and when each is a reasonable default.
Explain weight decay as L2 regularization applied during optimization.
Connect optimizers to backpropagation and overfitting controls.

What an Optimizer Does

Training a neural network is repeated minimization of a scalar loss over millions of parameters. Each step:

Forward pass — compute predictions (see Forward Propagation).
Backward pass — compute ∂loss/∂weight for every parameter.
Optimizer step — update weights using those gradients and hyperparameters.

Definition — Optimizer

An optimizer is an algorithm that uses gradients (and optionally past gradients or parameter statistics) to update model parameters so loss decreases. In PyTorch, optimizers implement zero_grad(), step(), and optional state per parameter.

Optimizer	Core Idea	Typical Use
`SGD`	Fixed learning rate × gradient	CV baselines, fine-tuning with momentum
`SGD + momentum`	Velocity-smoothed updates	ResNets, classical vision training
`Adam`	Per-parameter adaptive rates	Default for many NLP / tabular DL tasks
`AdamW`	Adam + decoupled weight decay	Transformers, modern default
`RMSprop`	Scale by running average of squared grads	RNNs, some legacy setups

PyTorch Optimizer Setup

Pass model.parameters() to the optimizer once. Reuse the same instance across epochs; only call zero_grad() before each backward pass.

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 128),
    nn.ReLU(),
    nn.Linear(128, 10),
)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)

for epoch in range(5):
    for x_batch, y_batch in train_loader:
        optimizer.zero_grad()
        logits = model(x_batch)
        loss = criterion(logits, y_batch)
        loss.backward()
        optimizer.step()

Volume 05 Bridge weight_decay in the optimizer is L2 regularization from Ridge — it penalizes large weights to fight overfitting.

Choosing an Optimizer in Practice

Start Here

AdamW(lr=1e-3) for most new projects
Log train and validation loss every epoch
Tune learning rate before exotic optimizers

When to Switch

SGD + momentum for image classification at scale
Lower LR + Adam when loss oscillates
Match optimizer to published baseline for reproducibility

Common Misconception: “Adam always beats SGD.”

Reality: Adam converges faster early but SGD with momentum often reaches better final accuracy on some vision tasks given enough tuning. Compare on your validation set.

Common Misconception: “You need a different optimizer object each epoch.”

Reality: Optimizers hold state (momentum buffers, Adam moments). Create once, reuse for the full run unless you change architecture or parameter groups.

Critical Mistake — Forgetting zero_grad()

Gradients accumulate in .grad tensors by default. Skipping optimizer.zero_grad() before loss.backward() poisons updates with stale gradients from prior batches.

Knowledge Check

Short Answer: What three calls form the core training step after forward pass? Answer: zero_grad, backward, step.
True/False: The optimizer computes loss values. Answer: False—the loss function does; the optimizer updates parameters from gradients.
Multiple Choice: weight_decay in AdamW primarily acts as: (a) batch norm, (b) L2 regularization, (c) dropout, (d) activation. Answer: (b).
Short Answer: Why pass model.parameters() to the optimizer? Answer: So it knows which tensors to update and can store per-parameter state.
True/False: Adam adapts learning rate per parameter. Answer: True.
Multiple Choice: First lecture on gradient application after loss: (a) Forward Propagation, (b) SGD, (c) Dropout, (d) Softmax. Answer: (b).
Short Answer: What happens if you skip zero_grad()? Answer: Gradients accumulate across steps, corrupting updates.
True/False: Optimizers require manually computed gradients in modern PyTorch. Answer: False—autograd computes them via backward().
Multiple Choice: Best default for many new DL projects: (a) no optimizer, (b) AdamW, (c) random search, (d) k-NN. Answer: (b).
Short Answer: Name one hyperparameter every optimizer shares. Answer: Learning rate (or equivalent step size).

Key Takeaways

Optimizers turn gradients into parameter updates.
PyTorch pattern: zero_grad → backward → step.
AdamW is a strong default; SGD+momentum still matters in vision.
Weight decay links optimization to regularization.
Next: Forward Propagation — how data flows through the network before gradients exist.

Trainer’s Guide

Hands-on idea: Train the same MLP on MNIST with SGD and AdamW; plot loss curves side by side for five epochs.

Discussion prompt: Why might production teams freeze the optimizer choice when reproducing a paper baseline?

What’s Next Optimizers need forward-computed activations and losses — continue to Forward Propagation.