SGD — Module 6.1 | Codextroop

How This Lesson Fits the Module

Backpropagation delivers gradients. Stochastic Gradient Descent (SGD) is the simplest rule for using them: step opposite the gradient, scaled by a learning rate.

Every adaptive optimizer (Adam) builds on this foundation. SGD with momentum remains the gold standard for many image models when tuned carefully.

Learning Objectives

By the end of this lesson, students should be able to:

Write the SGD update rule for a single parameter.
Contrast batch, mini-batch, and true stochastic gradient descent.
Configure torch.optim.SGD with momentum and weight decay.
Explain noise in mini-batch gradients as a regularizing effect.
Recognize when SGD outperforms adaptive methods on validation metrics.

The Update Rule

For parameter θ and loss L, one SGD step is:

θ ← θ − η ∇_θL

where η is the learning rate. With momentum, a velocity term accumulates past gradients for smoother descent.

Definition — Stochastic Gradient Descent

SGD estimates the full-dataset gradient using one sample or a mini-batch. The estimate is noisy but cheap, enabling training on datasets too large to load into memory at once.

Variant	Gradient Source	Updates per Epoch
Batch GD	Entire training set	1
Mini-batch SGD	One batch	⌈N / batch_size⌉
True SGD	Single example	N

PyTorch SGD with Momentum

import torch
import torch.nn as nn

model = nn.Linear(10, 2)
optimizer = torch.optim.SGD(
    model.parameters(),
    lr=0.05,
    momentum=0.9,        # dampens oscillation in ravines
    weight_decay=1e-4,   # L2 penalty each step
    nesterov=True,       # look-ahead gradient (often helps)
)

for x_batch, y_batch in train_loader:
    optimizer.zero_grad()
    loss = nn.CrossEntropyLoss()(model(x_batch), y_batch)
    loss.backward()
    optimizer.step()

SGD Strengths

Simple, well-understood dynamics
Strong final accuracy with tuning (vision)
Generalizes well with right batch size
Less memory than Adam state buffers

SGD Challenges

Sensitive to learning rate schedule
Slow progress on ill-conditioned losses
Requires more manual tuning than Adam
Noisy updates with tiny batches

Common Misconception: “SGD means one sample at a time.”

Reality: In deep learning, “SGD” almost always means mini-batch SGD. True single-example updates are rare on GPUs because vectorized batches are faster.

Common Misconception: “Higher momentum always helps.”

Reality: Momentum near 1.0 can overshoot minima. Common values: 0.9 for CNNs, 0.99 for some NLP setups. Validate on a held-out set.

Knowledge Check

Short Answer: SGD update direction? Answer: Opposite the gradient (descent).
True/False: Mini-batch SGD uses the full dataset per step. Answer: False.
Multiple Choice: momentum=0.9 primarily: (a) adds dropout, (b) smooths updates, (c) doubles batch size, (d) freezes weights. Answer: (b).
Short Answer: Symbol for learning rate in the update rule? Answer: eta (η).
True/False: weight_decay in SGD acts like L2 regularization. Answer: True.
Multiple Choice: Typical PyTorch “SGD” uses: (a) full batch only, (b) mini-batches, (c) no gradients, (d) Adam moments. Answer: (b).
Short Answer: Why is mini-batch noise sometimes beneficial? Answer: It can help escape sharp minima and add implicit regularization.
True/False: nesterov=True enables look-ahead gradient evaluation. Answer: True.
Multiple Choice: Adaptive per-parameter optimizer next: (a) Adam, (b) Dropout, (c) Weights, (d) Perceptron. Answer: (a).
Short Answer: One hyperparameter that most affects SGD convergence speed. Answer: Learning rate.

Key Takeaways

SGD: θ ← θ − η∇L on mini-batch gradients.
Momentum and Nesterov accelerate convergence in ravines.
Mini-batch SGD is the practical default meaning of “SGD.”
Tune learning rate and schedule before exotic tricks.
Next: Adam — adaptive learning rates per parameter.

Trainer’s Guide

Lab: Train CIFAR-10 with SGD+momentum vs Adam; compare final validation accuracy and wall-clock time.

Exercise: Students derive the SGD update for a single weight in a linear neuron by hand.

What’s Next SGD is the baseline — meet adaptive optimization in Adam.