Gradient Descent

Introduction: From Gradient to Algorithm

In supervised learning, a model has parameters (weights) w that map inputs to predictions. Training means finding w that minimizes a loss function L(w)—a scalar measuring how wrong the model is on data. You cannot try every possible w in a high-dimensional space; you need an iterative procedure.

Gradient descent is that procedure. At each step, compute the gradient ∇_wL—the vector of partial derivatives of the loss with respect to every parameter. The gradient points uphill (toward higher loss). To reduce loss, move in the opposite direction.

This lecture turns the gradient from a mathematical object into a training algorithm—the engine behind modern deep learning.

The Update Rule

Unpacking the symbols:

Component-wise view: If w = (w₁, w₂, …, w_n), then each parameter updates independently:

w_i ← w_i − α · (∂L / ∂w_i)

The gradient packages all these partial derivatives into one vector so we can write a single, compact update.

Why Subtract the Gradient?

The gradient ∇L points in the direction of steepest ascent—the direction that increases L fastest. Minimization requires the opposite direction: −∇L.

Geometric intuition: imagine standing on a hillside where height represents loss. The gradient points uphill. To reach the valley, walk downhill—opposite to the gradient. The learning rate α determines how far you stride in that downhill direction each step.

If you accidentally add the gradient (w ← w + α∇L), you climb uphill and loss increases. This is gradient ascent—useful for maximization problems, but wrong for training classifiers and regressors.

The Learning Rate α

The learning rate is the single most important hyperparameter in gradient descent. It scales every component of the gradient equally.

Unlike parameters w, the learning rate is typically not learned by gradient descent itself (though learning rate schedules and adaptive optimizers like Adam adjust effective step sizes per parameter). Engineers set α or choose a schedule: constant, step decay, cosine annealing, warmup-then-decay.

Estimating the Gradient: Batch, Mini-Batch, and SGD

The update rule assumes we can compute ∇L. But in ML, L is almost always an average loss over training data:

L(w) = (1/N) ∑_i=1^N L_i(w)

where L_i is the loss on the i-th training example. The true gradient is the average of per-example gradients. How we approximate that average defines three variants:

Mini-batch gradient descent is the workhorse of deep learning. A typical batch size ranges from 32 to 4096 depending on GPU memory and model size. The gradient estimate is:

∇L ≈ (1/B) ∑_{i ∈ batch} ∇L_i

This is an unbiased estimate of the true gradient when batches are sampled uniformly. The noise introduced by mini-batching can actually help escape poor regions—similar to shaking a ball out of a shallow dip.

The Loss Landscape: Local Minima and Beyond

Gradient descent follows local slope information. It has no global map of the loss surface. That creates well-known challenges:

• Local minima — Valleys that are low locally but not globally. Gradient descent can get stuck, returning a suboptimal solution.
• Saddle points — Points where the gradient is zero but the point is neither a minimum nor a maximum (like a mountain pass). Common in high dimensions; can slow training dramatically.
• Plateaus — Regions where the gradient is very small everywhere nearby. Progress stalls even though you are not at a minimum.

For simple convex problems (linear regression with MSE, logistic regression), the loss has a single global minimum—gradient descent is guaranteed to find it (with an appropriate α). Neural networks are non-convex: millions of local minima and saddle points exist.

Momentum: A Brief Preview

Plain gradient descent can zigzag in narrow valleys and crawl across flat regions. Momentum addresses this by accumulating a velocity vector that smooths updates over time.

v ← βv − α∇L,   w ← w + v

Here v is a velocity buffer and β ∈ [0, 1) is a momentum coefficient (often 0.9). The update remembers past gradient directions—like a ball rolling downhill, gaining speed in consistent directions and dampening oscillations across ravines.

Momentum is one of many extensions to vanilla gradient descent. The next lecture, Optimization, covers Adam, RMSprop, learning rate schedules, and other optimizers used in production training.

The Training Loop

Gradient descent is not a single formula—it is a loop embedded in a larger training procedure. Every PyTorch, TensorFlow, or JAX training script follows this skeleton:

Key engineering details often omitted from the formula but critical in practice:

• Zeroing gradients — Frameworks accumulate gradients by default. Call optimizer.zero_grad() before each backward pass or gradients from prior batches add up incorrectly.
• Evaluation mode — Disable dropout and batch-norm training behavior during validation (model.eval() in PyTorch).
• Gradient clipping — Cap gradient magnitude to prevent explosion in recurrent and transformer models (covered in later volumes).
• Early stopping — Halt training when validation loss stops improving to avoid overfitting.

PyTorch: Gradient Descent in Code

PyTorch hides the calculus behind two mechanisms: loss.backward() computes gradients via autograd, and optimizer.step() applies the update rule. Below is a high-level training loop for a simple classifier—not production code, but the structural template every engineer recognizes:

import torch
import torch.nn as nn
from torch.utils.data import DataLoader

# Model, loss, and optimizer (w is model.parameters(), alpha is lr)
model = nn.Linear(784, 10)                    # w: weights and biases
criterion = nn.CrossEntropyLoss()             # L: scalar loss
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)  # alpha = 0.01

train_loader = DataLoader(dataset, batch_size=64, shuffle=True)

for epoch in range(num_epochs):
    model.train()
    for inputs, labels in train_loader:     # mini-batch loop
        # Forward pass
        outputs = model(inputs)               # y_hat = f(w; x)
        loss = criterion(outputs, labels)     # L(w)

        # Backward pass: compute nabla_w L
        optimizer.zero_grad()                 # clear old gradients
        loss.backward()                       # autograd: chain rule

        # Update rule: w <- w - alpha * nabla_w L
        optimizer.step()

    # Validation (no gradient updates)
    model.eval()
    with torch.no_grad():
        val_loss = evaluate(model, val_loader)

Mapping code to math:

Switching from SGD to momentum is a one-line change: torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9). Switching to Adam: torch.optim.Adam(model.parameters(), lr=1e-3). The training loop structure stays identical—only the optimizer’s internal update rule changes.

Convergence and Stopping Criteria

When do you stop iterating? Common criteria:

• Fixed epochs — Train for a predetermined number of passes (e.g., 10, 50, 100). Simple but arbitrary.
• Loss threshold — Stop when L drops below a target. Rarely used alone because the target is unknown beforehand.
• Early stopping — Monitor validation loss; stop when it rises for several consecutive checks. Prevents overfitting.
• Gradient norm — Stop when ‘∇L’ is near zero, indicating a stationary point. Uncommon as a sole criterion because near-zero gradients also occur at saddles and plateaus.

In practice, engineers combine fixed epochs with early stopping and checkpointing: save the best validation model, then restore it after training ends.

Common Misconceptions

Quick Knowledge Check

Further Reading & References

Books

• Deep Learning — Goodfellow, Bengio, and Courville (2016). Chapter 8: Optimization for Training Deep Models. The definitive treatment of gradient descent variants.
• Pattern Recognition and Machine Learning — Christopher Bishop. Section 5.2: Gradient descent and error surfaces.

Video & Visual

• Gradient Descent, How Neural Networks Learn — 3Blue1Brown (YouTube). Visual intuition for the update rule on a 2D loss surface.
• Google Machine Learning Crash Course — Gradient descent module with interactive learning rate demonstrations.

Official Documentation

• PyTorch Optimizers — SGD, Adam, and all built-in update rules
• PyTorch Neural Networks Tutorial — Training loop walkthrough
• PyTorch Autograd — How backward() computes gradients