Validation Loop — Module 6.2

How This Lesson Fits the Module

Training loss tells you how well the model memorizes batches it just saw. A validation loop measures generalization on held-out data each epoch—the signal for early stopping, learning-rate schedules, and checkpoint selection.

Volume 05 introduced validation splits for sklearn models. Deep learning runs validation inside the training script, often every epoch.

Prior Lesson You built a training loop that updates weights. Validation reuses the same forward pass but disables gradient computation and weight updates.

Learning Objectives

By the end of this lesson, students should be able to:

Write a validate() function with model.eval() and torch.no_grad().
Compute validation loss and accuracy from a separate DataLoader.
Explain why validation data must never flow through loss.backward().
Compare train vs validation curves to spot overfitting.
Choose metrics aligned with the task (accuracy, F1, perplexity).
Integrate validation into the outer epoch loop after training.

Train vs Validation Behavior

Validation is inference-mode evaluation on data the optimizer never sees. Dropout is off; batch normalization uses running statistics. No optimizer step runs.

Aspect	Training loop	Validation loop
Mode	`model.train()`	`model.eval()`
Gradients	Enabled	`torch.no_grad()`
Shuffle	Usually yes	Usually no
Optimizer	`step()` called	Not used
Purpose	Update weights	Estimate generalization

Validation Function in PyTorch

Mirror the training loop structure but accumulate metrics instead of calling the optimizer. Divide totals by the number of batches for average loss.

def validate(model, loader, criterion, device):
    model.eval()
    running_loss = 0.0
    correct = 0
    total = 0

    with torch.no_grad():
        for inputs, targets in loader:
            inputs, targets = inputs.to(device), targets.to(device)
            outputs = model(inputs)
            loss = criterion(outputs, targets)
            running_loss += loss.item()

            preds = outputs.argmax(dim=1)
            correct += (preds == targets).sum().item()
            total += targets.size(0)

    avg_loss = running_loss / len(loader)
    accuracy = correct / total
    return avg_loss, accuracy

for epoch in range(1, num_epochs + 1):
    train_loss = train_one_epoch(model, train_loader, criterion, optimizer, device)
    val_loss, val_acc = validate(model, val_loader, criterion, device)
    print(f"Epoch {epoch:03d} | train={train_loss:.4f} | val={val_loss:.4f} | acc={val_acc:.3f}")

Critical Mistake — Tuning on the Test Set

Validation guides hyperparameter and architecture choices. The test set is evaluated once at the end (next lesson). Peeking at test metrics each epoch and picking the best epoch is the same leakage Volume 05 warned about—just inside PyTorch.

Validation DataLoader Setup

Use shuffle=False for reproducible metrics. Batch size can be larger than training since no gradients are stored—limited only by GPU memory for activations.

val_loader = DataLoader(
    val_dataset,
    batch_size=256,
    shuffle=False,
    num_workers=4,
    pin_memory=torch.cuda.is_available(),
)

Reading Train–Val Curves

Train loss down, validation loss flat or rising—classic overfitting. Both rising—learning rate too high or broken pipeline. Both flat—underfitting or converged. Log both every epoch; plots beat single numbers.

Engineering Habit — One Metric for Decisions

Pick a primary validation metric (e.g. val_loss or macro-F1) for checkpointing and early stopping. Secondary metrics are for dashboards, not competing stop rules.

Knowledge Check

Short Answer: Why use torch.no_grad() in validation? Answer: Skips gradient computation, saving memory and time.
True/False: Validation should call optimizer.step(). Answer: False—weights must not update on validation data.
Multiple Choice: model.eval() affects: (a) dropout, (b) learning rate, (c) DataLoader shuffle only. Answer: (a) and batch-norm behavior.
Short Answer: Should validation batches be shuffled? Answer: Usually no—order does not affect metrics and aids reproducibility.
Short Answer: What gap suggests overfitting? Answer: Training loss improves while validation loss worsens.
True/False: Validation loss can guide learning-rate schedules. Answer: True—e.g. ReduceLROnPlateau (later lesson).
Multiple Choice: Validation runs: (a) before training each epoch, (b) after training each epoch, (c) only once. Answer: (b) is typical.
Short Answer: Why can val batch size exceed train batch size? Answer: No gradient storage means lower memory per sample.
Short Answer: What is the validation set for in DL training scripts? Answer: Model selection and monitoring generalization during training.
Multiple Choice: Using test set every epoch to pick best model is: (a) best practice, (b) data leakage, (c) required. Answer: (b).

Key Takeaways

Validation = eval() + no_grad() + no optimizer; same forward pass as training.
Run validation each epoch; compare curves to diagnose fit.
Reserve the test set for a single final evaluation.
Next: Test Loop—the one-time sign-off pass.

Trainer’s Guide

Hands-on idea: Train a small CNN for 30 epochs without regularization; plot train vs val loss. Ask students when they would have stopped.

Discussion prompt: How is validation in a DL script different from cross_val_score in sklearn?

What’s Next After validation-driven training finishes, run the Test Loop exactly once on held-out data.