Loss Functions — Module 6.1

How This Lesson Fits Module 6.1

Loss functions close the first 12-lecture foundation sequence: inputs flow through layers, weights and bias terms produce logits, activations interpret them, and the loss tells the optimizer what must improve. This prepares students for forward propagation, backpropagation, and optimizers later in Module 6.1.

Learning Objectives

By the end of this lesson, students should be able to:

Define a loss function as the scalar objective minimized during training.
Match common losses to regression, binary classification, multiclass classification, and multilabel tasks.
Explain why PyTorch classification losses often expect logits.
Distinguish training loss from evaluation metrics.
Implement a basic training step with loss and backpropagation.
Recognize loss-target shape and dtype mistakes.

Definition

A loss function converts model outputs and true targets into a scalar penalty that measures how wrong the model is for the training objective.

The Objective Drives Learning

A neural network does not improve because it knows accuracy directly. It improves because the loss produces gradients. The choice of loss defines what errors matter and how strongly. Regression losses penalize numeric distance; classification losses reward high score on the correct class; imbalance-aware variants change the penalty weighting. Good training starts by matching output shape, target format, and loss function.

Task	Output expected	Common loss
Regression	Continuous value	`nn.MSELoss`, `nn.L1Loss`, Huber
Binary classification	Raw logit	`nn.BCEWithLogitsLoss`
Multiclass classification	Raw logits `[N,C]`	`nn.CrossEntropyLoss`
Multilabel classification	Independent logits `[N,C]`	`nn.BCEWithLogitsLoss`
Imbalanced classes	Task dependent	Weighted CE, focal loss, sampling strategies

PyTorch Practice

A training step computes logits, computes loss, clears old gradients, backpropagates, and updates parameters.

import torch
from torch import nn

model = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 3))
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2)
loss_fn = nn.CrossEntropyLoss()

x = torch.randn(16, 10)
y = torch.randint(0, 3, (16,))

logits = model(x)
loss = loss_fn(logits, y)

optimizer.zero_grad()
loss.backward()
optimizer.step()

print(loss.item())

Loss vs Metric

Loss

Optimized directly
Must be differentiable or gradient-friendly
Used every training step

Metric

Reports task success
May be nondifferentiable
Used for validation and model selection

Both

Should be monitored
Can disagree under imbalance
Need clear train/validation separation

Strengths and Tradeoffs

Useful because

Turns model errors into gradients for learning.
Can encode task priorities and class weights.
Provides a comparable training curve over time.

Watch for

Wrong loss can optimize the wrong behavior perfectly.
Loss values are not always human-interpretable metrics.
Shape or dtype mismatches can silently distort training or fail late.

How It Flows

1. Forward

Model produces outputs from input tensors.

2. Compare

Loss function compares outputs with targets.

3. Reduce

Per-example penalties become a scalar.

4. Backward

Autograd computes parameter gradients.

5. Optimize

The optimizer updates weights and bias terms.

Common Misconception

Do not choose a loss from the activation name alone. Choose it from the task and target format: one class among many uses cross-entropy; multiple independent labels use binary cross-entropy with logits.

Knowledge Check

Short Answer: What does a loss function output? Answer: A scalar penalty/objective.
True/False: The optimizer uses loss gradients to update parameters. Answer: True.
Multiple Choice: Multiclass classification commonly uses: (a) CrossEntropyLoss, (b) MSELoss always, (c) no targets. Answer: (a).
Short Answer: Binary logits pair with which loss? Answer: BCEWithLogitsLoss.
True/False: Accuracy is always the training loss. Answer: False.
Short Answer: What must happen before loss.backward() in a standard step? Answer: Compute model outputs and loss; usually clear old gradients before backward.
Multiple Choice: Regression often uses: (a) MSE, (b) softmax CE only, (c) argmax loss. Answer: (a).
Short Answer: Why monitor validation loss? Answer: To detect generalization and overfitting behavior.
True/False: Target dtype can matter for PyTorch losses. Answer: True.
Short Answer: What does weighted loss help with? Answer: Class imbalance or unequal error costs.

Key Takeaways

Loss functions define what training is trying to minimize.
Output shape, target encoding, and loss must agree.
Loss is optimized; metrics explain whether the model is useful.
Next, Forward Propagation follows data through the network step by step.

Trainer’s Guide

Hands-on idea: Give students four mini task descriptions and ask them to choose output dimensions, target dtype, and loss before coding.

Discussion prompt: When could a lower training loss produce a worse product decision?

Recap: A loss function turns predictions into the gradients that train the network, so choosing it correctly is foundational. Continue with Forward Propagation.