Backpropagation — Module 6.1

How This Lesson Fits the Module

Forward propagation computes predictions. Backpropagation computes how each weight contributed to the loss—efficiently, via the chain rule.

PyTorch’s loss.backward() runs backprop automatically. Understanding the idea prepares you for SGD and debugging vanishing gradients in deep stacks.

Learning Objectives

By the end of this lesson, students should be able to:

State the chain rule role in layered networks.
Explain backward flow from loss to each parameter.
Use loss.backward() and inspect .grad in PyTorch.
Identify why ReLU and depth affect gradient magnitude.
Connect backprop outputs to optimizer steps.

The Chain Rule in Layers

If L depends on y, and y = f(x), then dL/dx = (dL/dy)(dy/dx). In a network, each layer is one link in a long chain from loss to input.

Definition — Backpropagation

Backpropagation is an algorithm that applies the chain rule layer by layer, reusing intermediate gradients to compute partial derivatives of the loss with respect to every trainable parameter in one backward sweep. It is the standard way to train deep networks because it shares computation across parameters.

Pass	Direction	Computes
Forward	Input → output	Activations, loss value
Backward	Loss → input	∂L/∂W, ∂L/∂b for each layer
Optimizer step	N/A	Parameter update from gradients

PyTorch Autograd in Practice

import torch
import torch.nn as nn

torch.manual_seed(0)
x = torch.randn(4, 3, requires_grad=False)
y = torch.tensor([0, 1, 2, 0])
model = nn.Linear(3, 2)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

optimizer.zero_grad()
logits = model(x)
loss = criterion(logits, y)
loss.backward()                    # populates .grad on weights and bias

print("loss:", loss.item())
print("weight grad shape:", model.weight.grad.shape)  # (2, 3)
optimizer.step()

Gradient Flow and Depth

Deep networks multiply many Jacobian terms. Saturating activations (sigmoid, tanh) can shrink gradients (vanishing); large weights can explode them. ReLU, residual connections, and batch normalization help keep signals trainable.

Volume 05 Bridge Backprop minimizes empirical loss on the training set. Monitor validation loss to catch overfitting while gradients still flow.

Common Misconception: “You must implement backprop by hand in PyTorch.”

Reality: Autograd differentiates operations in the computation graph. Hand-derived gradients are for learning and custom CUDA kernels—not daily training loops.

Common Misconception: “backward() updates weights.”

Reality: backward() only fills .grad. optimizer.step() applies the update. See Optimizers.

Critical Mistake — Detached Tensors

Calling .detach() or using torch.no_grad() during training blocks gradients. Symptoms: .grad is None or loss flatlines. Reserve no_grad for validation and inference.

Knowledge Check

Short Answer: What mathematical rule powers backprop? Answer: The chain rule.
True/False: backward() modifies weights directly. Answer: False—optimizer.step() does.
Multiple Choice: After backward(), gradients live in: (a) .data, (b) .grad, (c) .bias, (d) optimizer state only. Answer: (b).
Short Answer: Why are forward activations cached? Answer: Needed to compute gradients in the backward pass.
True/False: ReLU can pass gradients unchanged for positive inputs. Answer: True.
Multiple Choice: Vanishing gradients are worsened by: (a) ReLU, (b) saturated sigmoid, (c) residual skip, (d) batch norm. Answer: (b).
Short Answer: What call triggers autograd backward? Answer: loss.backward() (or torch.autograd.grad).
True/False: Inputs x usually need requires_grad=True for standard supervised training. Answer: False—parameters do.
Multiple Choice: Next lecture on applying gradients: (a) SGD, (b) Perceptron, (c) Input Layer, (d) Softmax. Answer: (a).
Short Answer: Name one symptom of blocked gradients. Answer: None .grad, flat loss, or no learning.

Key Takeaways

Backprop applies the chain rule efficiently from loss to parameters.
backward() fills .grad; optimizer.step() updates weights.
Depth and activation choice strongly affect gradient health.
PyTorch autograd handles derivatives for standard layers.
Next: SGD — the classic gradient update rule.

Trainer’s Guide

Micro-demo: Two-layer net on a toy 2D dataset; print weight.grad norms after one backward pass.

Discussion prompt: Why did deep learning take off after backprop + GPUs, not after perceptrons alone?

What’s Next Gradients are ready — apply them with SGD.