Early Stopping — Module 6.2

How This Lesson Fits the Module

Neural networks often keep improving training loss while validation loss rises—overfitting. Early stopping halts training when validation stops improving, saving compute and returning weights from the best epoch automatically.

Paired with checkpoints, early stopping is the standard regularization-free guardrail in production training pipelines.

Prior Lesson Checkpoints persist the best model. Early stopping defines when to stop searching for a better one.

Learning Objectives

By the end of this lesson, students should be able to:

Implement patience-based early stopping on a validation metric.
Configure min_delta to ignore noise-level improvements.
Restore best weights after stopping (not last epoch weights).
Choose whether to minimize loss or maximize accuracy/F1.
Explain relationship between early stopping and L2 regularization.
Log stop reason and best epoch for experiment tracking.

Early Stopping Logic

Track the best validation score. Each epoch without sufficient improvement increments a counter. When counter exceeds patience, stop and reload the best checkpoint.

Hyperparameter	Typical value	Effect
`patience`	5–20 epochs	Epochs to wait after last improvement
`min_delta`	1e-4 (loss) or 0.001 (acc)	Minimum change to count as improvement
Monitor	`val_loss` or `val_acc`	Metric driving the stop rule
`mode`	`"min"` or `"max"`	Loss vs accuracy direction

PyTorch Early Stopping Class

PyTorch has no built-in callback like Keras. A small class keeps training scripts clean.

class EarlyStopping:
    def __init__(self, patience=10, min_delta=0.0, mode="min"):
        self.patience = patience
        self.min_delta = min_delta
        self.mode = mode
        self.best = float("inf") if mode == "min" else float("-inf")
        self.counter = 0
        self.should_stop = False

    def step(self, metric):
        improved = (
            metric < self.best - self.min_delta if self.mode == "min"
            else metric > self.best + self.min_delta
        )
        if improved:
            self.best = metric
            self.counter = 0
            return True  # signal caller to save checkpoint
        self.counter += 1
        if self.counter >= self.patience:
            self.should_stop = True
        return False

early_stop = EarlyStopping(patience=7, min_delta=1e-4, mode="min")

for epoch in range(1, num_epochs + 1):
    train_one_epoch(...)
    val_loss, val_acc = validate(...)

    if early_stop.step(val_loss):
        save_checkpoint("checkpoints/best.pt", model, optimizer, epoch, val_loss)

    if early_stop.should_stop:
        print(f"Early stop at epoch {epoch}. Best val_loss={early_stop.best:.4f}")
        break

# Reload best weights for test evaluation
ckpt = torch.load("checkpoints/best.pt", map_location=device)
model.load_state_dict(ckpt["model_state_dict"])

Critical Mistake — Stopping Without Restoring Best Weights

After early stop, the model object still holds last-epoch weights—often worse than the best. Always load_state_dict from the best checkpoint before test or export.

Patience vs Dataset Size

Noisy validation curves on small sets need higher patience or larger min_delta. Large stable datasets can use patience 3–5. Plot validation before choosing—do not copy Kaggle defaults blindly.

Engineering Habit — One Stop Rule

Do not early-stop on train loss. Validation (or a dedicated eval split) is the only legitimate monitor for stopping.

Knowledge Check

Short Answer: What does patience control? Answer: How many epochs without improvement before stopping.
True/False: Early stopping reduces overfitting risk. Answer: True—by halting before validation degrades further.
Multiple Choice: For val_acc, mode should be: (a) min, (b) max, (c) either. Answer: (b).
Short Answer: What is min_delta for? Answer: Ignoring tiny fluctuations that are not real improvements.
Short Answer: After break, which weights should you deploy? Answer: Best validation checkpoint, not last epoch.
True/False: Early stopping replaces the need for a test set. Answer: False—still need final test evaluation.
Multiple Choice: Counter resets when: (a) train loss drops, (b) monitored metric improves enough, (c) every epoch. Answer: (b).
Short Answer: How is early stopping similar to L2? Answer: Both limit effective model complexity / training duration.
Short Answer: Why log best epoch? Answer: Audit trail and comparison across experiments.
Multiple Choice: Patience=1 means: (a) stop after first non-improving epoch, (b) never stop, (c) train one epoch total. Answer: (a).

Key Takeaways

Early stopping watches validation metrics with patience and min_delta.
Save and reload best weights—never deploy the last epoch by default.
Tune patience to validation noise and dataset size.
Next: Learning Rate Scheduler—adapt LR during training.

Trainer’s Guide

Hands-on idea: Run the same model with and without early stopping; compare test accuracy and total GPU time.

Discussion prompt: Can early stopping hide underfitting? When would you disable it?

What’s Next Combine stopping with dynamic learning rates in Learning Rate Scheduler.