Optimization

How This Lesson Fits the Module

The previous lecture, Gradient Descent, introduced the core update rule—moving parameters opposite the gradient to reduce loss. That algorithm is correct but incomplete. Real models have millions of parameters, noisy minibatch gradients, curved loss surfaces, and training budgets measured in epochs, not calculus homework problems.

Optimization is the capstone of Module 2.2. Here we zoom out from a single update step to the full engineering of training: which optimizer to choose, how to schedule the learning rate, how regularization reshapes the landscape you descend, and why deep networks live in a non-convex world where classical guarantees no longer apply. This lecture ties together derivatives through gradient descent into the practical toolkit that every practitioner uses—and prepares you for Module 2.3, where probability explains the noise, uncertainty, and data distributions that make optimization stochastic in the first place.

Learning Objectives

By the end of this lesson, students should be able to:

Distinguish convex from non-convex optimization and explain why neural network training is fundamentally non-convex.
Describe the loss landscape geometrically—valleys, saddles, flat minima—and connect curvature to the Hessian and eigenvalues.
State the SGD update rule and explain the role of minibatch stochasticity in training.
Compare SGD, RMSprop, and Adam: what each remembers, how each scales learning rates, and when each is a reasonable default.
Implement learning rate schedules conceptually—constant, decay, warmup, and cosine annealing—and diagnose too-large vs too-small learning rates.
Explain how L1, L2, and early stopping modify the effective optimization problem and interact with gradient-based methods.
Recap the calculus module arc from scalar derivatives to multivariate gradient descent.
Articulate how optimization connects to probability: expected risk, noisy gradients, and uncertainty over data.

Introduction: From Calculus to Training Loops

When you call model.fit() or run a PyTorch training loop, you are not “doing calculus” in the abstract. You are solving an optimization problem: find parameters θ that minimize a loss function L(θ) built from data, architecture, and regularization terms.

Module 2.2 gave you the language for that problem. Derivatives measure sensitivity. Partial derivatives extend the idea to many variables. The chain rule decomposes gradients through composed functions—the backbone of backpropagation. The gradient collects partials into the direction of steepest ascent; gradient descent steps in the opposite direction. This lecture asks the questions practitioners face after that foundation is in place:

Is the surface we are descending well-behaved (convex), or riddled with traps (non-convex)?
Should every parameter share one learning rate, or adapt per coordinate?
How should the step size change over thousands of iterations?
How does regularization alter the landscape we optimize?

Optimization is where calculus meets engineering. The math tells you what a gradient is; optimization tells you how to use it reliably at scale.

The Loss Landscape

For a model with d parameters, the loss landscape is the graph of L(θ) over ℝ^d. You cannot visualize ℝ^1,000,000, but the metaphor is indispensable: training is a walk downhill on a high-dimensional terrain.

Definition — Loss Landscape

Given a loss function L : ℝ^d → ℝ and parameter vector θ ∈ ℝ^d, the loss landscape is the hypersurface { (θ, L(θ)) }. Its geometry—slopes, curvature, ridges, basins—determines how gradient-based methods behave.

Local Structure: Gradient and Hessian

At any point θ, the gradient ∇L(θ) (covered in Gradient) points uphill along the steepest direction. The Hessian H(θ) is the matrix of second partial derivatives:

H_ij = ∂²L / ∂θ_i∂θ_j

The Hessian describes curvature. Its eigenvalues classify critical points where ∇L = 0:

Hessian Eigenvalues at Critical Point	Classification	Landscape Shape
All positive	Local minimum	Bowl opening upward
All negative	Local maximum	Bowl opening downward
Mixed signs	Saddle point	Rising in some directions, falling in others

Linear Algebra ConnectionHessian eigenvalues generalize the second-derivative test from single-variable calculus. The spectral intuition from Module 2.1 eigenvalues applies directly: eigenvectors of H are principal curvature directions of the loss surface.

What Practitioners See in Low Dimensions

Researchers often project the landscape into two dimensions—for example, plotting L along two random directions in parameter space—to reveal multiple valleys, saddle ridges, and wide flat basins. These plots are not the full ℝ^d picture, but they explain why training trajectories depend on initialization and optimizer choice.

Sharp Minimum

High curvature around the minimum
Small perturbations to θ increase loss quickly
Often associated with poorer generalization in some studies
SGD noise may help avoid overly sharp basins

Flat Minimum

Low curvature in a wide neighborhood
Parameters can vary without large loss increase
Often linked to robustness and generalization
Large-batch methods may converge to sharper regions

Common Misconception: “Non-convex means gradient descent will get stuck in bad local minima.”

Reality: In heavily overparameterized neural networks, many local minima have similar loss values, and saddles—not local minima—are often the bigger obstacle. Stochastic gradients, momentum, and adaptive methods provide escape mechanisms that pure gradient descent on a clean convex function does not need.

Convex vs Non-Convex Optimization

Definition — Convex Function

A function f : ℝ^d → ℝ is convex if for all x, y and all θ ∈ [0, 1]:

f(θx + (1 − θ)y) ≤ θf(x) + (1 − θ)f(y)

Equivalently, the line segment between any two points on the graph lies on or above the graph. A convex optimization problem minimizes a convex f over a convex feasible set. Every local minimum is a global minimum.

Why Convexity Matters

Convex problems are the gold standard of optimization theory. Gradient descent with an appropriate step-size schedule converges to the global optimum. Linear regression with squared error (without deep feature compositions), logistic regression, SVMs with convex hinge losses, and Lasso objectives are convex or nearly so in their parameters.

The landscape has no spurious local minima. If you descend, you succeed. Analysis is clean; hyperparameters are forgiving.

Why Deep Learning Is Non-Convex

A neural network composes nonlinear functions: matrix multiplications, activations, normalization layers. The loss L(θ) is a high-degree composition of these operations. Composition destroys convexity.

Consider two parameter settings that both achieve low loss. Their average in parameter space is generally not a good model—the loss at the average can be far higher. That failure of the chord-above-graph property is non-convexity in action.

Convex Setting

One global minimum (possibly a flat manifold)
Gradient descent guarantees (with right η)
Loss landscape is a single bowl
Examples: linear/logistic regression, convex SVM

Non-Convex Setting (Deep Learning)

Many critical points: minima, saddles, plateaus
No global convergence guarantee from arbitrary init
Symmetries create equivalent solutions (permutation of neurons)
Heuristics—Adam, schedules, regularization—carry the day

Example — Mean Squared Error Is Convex in Weights (Linear Model)

For linear prediction ŷ = w^Tx and loss L(w) = ||y − Xw||², the Hessian is 2X^TX, which is positive semidefinite. The problem is convex in w. Add a two-layer ReLU network and the same MSE loss is no longer convex in all weights—the landscape folds and branches.

Gradient Descent Recap and Stochastic Variants

Before comparing optimizers, fix the baseline from Gradient Descent.

Batch Gradient Descent

Given loss L(θ) computed on the full dataset, iterate:

θ_t+1 = θ_t − η ∇L(θ_t)

where η > 0 is the learning rate. Each step uses the true gradient. Accurate but expensive when data is large.

Stochastic Gradient Descent (SGD)

Approximate the gradient with a random minibatch B_t of size m:

θ_t+1 = θ_t − η ∇L_{B_t}(θ_t)

where L_B is the loss on the minibatch. The update is unbiased in expectation: E[∇L_B(θ)] = ∇L(θ) when batches are sampled uniformly. The variance of the estimate injects noise—harmful for precision, often helpful for exploration.

SGD is not a different algorithm philosophically; it is gradient descent with a noisy gradient estimator. That noise is the first bridge to Module 2.3: training is inherently stochastic, and understanding randomness is as important as understanding derivatives.

Optimizer Overview: SGD, RMSprop, and Adam

Vanilla SGD uses one global learning rate η for every parameter. In practice, different coordinates of θ have gradients of wildly different magnitudes—think of bias terms vs deep layer weights, or sparse vs dense features. Adaptive optimizers rescale each parameter’s update using statistics accumulated over past iterations.

SGD with Momentum

Momentum smooths updates by accumulating a velocity vector:

v_t+1 = βv_t + ∇L_{B_t}(θ_t)
θ_t+1 = θ_t − ηv_t+1

with β ∈ [0, 1), typically 0.9. Momentum accelerates progress along consistent directions and dampens oscillation in ravines—where the gradient zigzags across a narrow valley while making slow progress along the valley floor.

RMSprop

RMSprop maintains an exponential moving average of squared gradients per parameter:

s_t+1 = ρs_t + (1 − ρ)(∇L_B)²
θ_t+1 = θ_t − η ∇L_B / (√s_t+1 + ε)

Division is element-wise. Large historical squared gradients shrink the effective step; small ones enlarge it. RMSprop adapts to varying curvature across coordinates—useful when the landscape is ill-conditioned.

Adam (Adaptive Moment Estimation)

Adam combines momentum (first moment) with RMSprop-style scaling (second moment):

m_t = β₁m_t−1 + (1 − β₁)g_t
v_t = β₂v_t−1 + (1 − β₂)g_t²
&hat;m_t = m_t / (1 − β₁^t) &hat;v_t = v_t / (1 − β₂^t)
θ_t+1 = θ_t − η &hat;m_t / (√&hat;v_t + ε)

where g_t = ∇L_{B_t}(θ_t). Defaults β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸ work surprisingly often. Bias correction (&hat;m, &hat;v) fixes the zero-initialization of moments in early steps.

Optimizer	What It Remembers	Per-Parameter Scaling	Typical Use Case
SGD (+ momentum)	Velocity (optional)	Uniform η (unless manual tuning)	CV models, fine-tuning with tuned η, some large-scale training when generalization matters
RMSprop	EMA of squared gradients	η / RMS(grad)	RNNs, non-stationary objectives; less common as default today
Adam	First and second moments	η × momentum / RMS(grad)	Default starting point for transformers, GANs, rapid prototyping

Practical Note

No optimizer wins everywhere. Adam converges fast early but SGD+momentum with careful tuning sometimes generalizes better on vision tasks. Modern frameworks also decouple weight decay from L2 regularization when using AdamW. Treat optimizer choice as an empirical question—but understand the mechanics so hyperparameter changes are intentional, not superstition.

Learning Rate Schedules

The learning rate η is the single most important hyperparameter in gradient-based training. Too large: loss spikes, divergence, NaNs. Too small: glacial progress, trapping in wide flat regions. Fixed η rarely suffices for full training runs.

Common Schedule Families

Schedule	Formula (schematic)	Behavior
Constant	η_t = η₀	Simple baseline; often combined with other techniques
Step decay	η_t = η₀ · γ^{⌊t / k⌋}	Drop by factor γ every k epochs; classic in ResNet training
Exponential decay	η_t = η₀ e^−λt	Smooth continuous decrease
Cosine annealing	η_t = η_min + ½(η₀ − η_min)(1 + cos(πt/T))	Gradual slowdown, popular in transformers and vision
Warmup	η_t linearly rises from 0 to η₀ over W steps	Stabilizes early training for large models and Adam; then decay

1. Warmup — Small steps while moment estimates stabilize → 2. Peak η — Fast learning in favorable regions → 3. Decay — Fine-grained convergence into a basin

Diagnostic — Learning Rate Too High vs Too Low

Too high: Training loss oscillates or explodes; validation loss diverges; gradients may overflow to Inf/NaN.

Too low: Loss decreases monotonically but plateaus far above known benchmarks; model underfits even with adequate capacity; increasing η by 2–10× may jump-start progress.

Just right: Steady decrease with occasional noise (especially SGD); validation tracks training without persistent gap growth early on.

Regularization and Optimization

Regularization is often taught as a separate topic from optimization, but in practice they are inseparable. Regularization defines the objective you optimize; the optimizer searches for minima of that objective.

Regularized Objective

L_total(θ) = L_data(θ) + λ R(θ)

L_data measures fit (e.g., cross-entropy, MSE). R(θ) penalizes complexity. λ ≥ 0 controls trade-off. Gradient descent minimizes L_total, not L_data alone.

L2 Regularization (Weight Decay)

R(θ) = ||θ||². Adds 2λθ to the gradient—each update shrinks weights toward zero. Geometrically, L2 pulls solutions toward smaller norms, encouraging smoother functions and wider basins. In AdamW, weight decay is applied directly to weights rather than mixed into the gradient of L2, which changes training dynamics slightly but meaningfully.

L1 Regularization

R(θ) = ||θ||₁. Promotes sparsity: many coordinates driven to exactly zero. The gradient is discontinuous at zero, so subgradient methods apply. Landscape becomes non-smooth along axes—optimization must tolerate corners.

Early Stopping

Not a penalty term, but a stopping rule: halt when validation loss stops improving. Equivalent to limiting optimization budget and preferring solutions reachable early—often smoother and better generalizing. You are choosing a point on the optimization path, not only the endpoint.

Without Regularization

Optimizer may drive L_data very low on training set
Sharp minima, large weights possible
High variance solutions; overfitting risk

With Regularization

Landscape reshaped; some valleys deepened, others raised
Gradient includes penalty terms guiding toward simpler θ
Optimization and generalization goals partially aligned

Chain Rule ConnectionBackpropagation computes ∇L_total by chaining partial derivatives through every layer (Chain Rule). Adding L2 changes the final gradient at each weight; the backward pass structure is unchanged.

Putting It Together: A Training Loop Mental Model

A single training epoch, abstracted:

Sample a minibatch B from the data distribution (randomness enters here).
Forward pass compute predictions and L_total(θ).
Backward pass compute gradients via backpropagation (partials + chain rule).
Optimizer step update θ using SGD, Adam, or variant; apply weight decay if configured.
Schedule adjust η according to step count or epoch.
Validate periodically; early-stop if configured.

Every step rests on Module 2.2 calculus. Every random choice previews Module 2.3 probability.

Module 2.2 Recap: From Derivatives to Descent

This lecture closes the calculus arc you built across Module 2.2:

Derivatives — instantaneous rate of change; sensitivity of a scalar output to a scalar input.
Partial Derivatives — holding other variables fixed; the building blocks of multivariate calculus.
Chain Rule — gradients through composed functions; the engine of backpropagation.
Gradient — the vector of partials pointing uphill; ∇L tells you how to change θ to increase loss.
Gradient Descent — iterate θ ← θ − η∇L; the foundational optimization algorithm.
Optimization (this lecture) — landscapes, convexity, stochastic methods, adaptive optimizers, schedules, and regularization—the full training picture.

You now possess the calculus toolkit behind modern AI training. When a loss curve misbehaves, you can ask: Is η wrong? Is the landscape ill-conditioned? Is the gradient estimate too noisy? Is regularization too weak? Those are optimization questions with calculus answers.

Bridge to Module 2.3 — Probability & Statistics

Calculus told you how to respond to a gradient. Probability tells you what you are averaging over.

Minibatch SGD approximates an expected gradient over the data distribution. Generalization asks whether low training loss implies low expected loss on unseen data—a question about distributions, not derivatives alone. Cross-entropy loss is derived from maximum likelihood; weight initialization schemes assume certain distributional properties; batch normalization estimates running mean and variance.

Module 2.3 — Probability & Statistics — begins with Probability, then builds through conditional probability, Bayes’ theorem, descriptive statistics, variance, and the Gaussian distribution. Together with Module 2.2, you will understand both how models learn (optimization) and what they learn from (data as samples from distributions).

Knowledge Check

Short Answer: Define a convex function and one practical consequence for gradient-based optimization. Answer: Chord lies above the graph; any local minimum is global, so GD cannot get stuck in a worse basin.
Short Answer: Why is deep-net training non-convex even with MSE loss? Answer: Composition of nonlinear layers makes L(θ) non-convex in the parameters.
Short Answer: Batch GD vs SGD: why does SGD inject noise, and when can that help? Answer: Minibatches estimate the full gradient; noise can escape sharp minima / saddles.
Short Answer: In Adam, what do the first and second moment estimates represent? Answer: Running mean of gradients (momentum) and running mean of squared gradients (adaptive scale).
Short Answer: Step decay vs cosine annealing; when is warmup needed? Answer: Step drops LR on a schedule; cosine smoothly anneals; warmup stabilizes large-batch / high-LR starts.
Short Answer: How does L2 regularization change the gradient and preferred solutions? Answer: Adds 2λθ to the gradient; prefers smaller weights (weight decay).
Short Answer: What is a saddle in Hessian eigenvalues, and why are saddles common in high-D? Answer: Mixed positive and negative curvature; most critical points in high-D are saddles, not local minima.
Short Answer: How does minibatch training connect to expectation over the data distribution? Answer: The minibatch gradient is an unbiased (noisy) estimate of E[∇ℓ] over the data distribution (Module 2.3).
True/False: RMSprop and Adam adapt per-parameter step sizes using accumulated gradient statistics. Answer: True.
Multiple Choice: Early stopping primarily: (a) increases training loss, (b) regularizes by halting when validation error rises, (c) replaces the Hessian, (d) removes the learning rate. Answer: (b).

Key Takeaways

The loss landscape is L(θ) over parameter space; its valleys, saddles, and curvature (Hessian) govern training dynamics.
Convex problems have no spurious local minima; deep learning is non-convex but heuristic optimizers work remarkably well in practice.
SGD uses noisy minibatch gradients; RMSprop and Adam adapt per-parameter step sizes using accumulated gradient statistics.
Learning rate schedules—warmup, decay, cosine annealing—are essential for stable convergence at scale.
Regularization (L1, L2, early stopping) defines the objective being optimized; it is not separate from the training loop.
Module 2.2 flows: derivatives → partials → chain rule → gradient → gradient descent → optimization engineering.
Module 2.3 adds probability: data distributions, expectation, and uncertainty complete the picture of learning from data.