Test Loop — Module 6.2

How This Lesson Fits the Module

The test loop is your sealed final exam. You load the best checkpoint chosen by validation, run one evaluation on the test set, and report those numbers in papers, dashboards, and release reviews. No hyperparameter changes afterward.

Volume 05’s testing-set discipline applies unchanged—deep learning just makes it easier to cheat by accident if test evaluation lives in the same script as training.

Prior Lessons Training updates weights; validation guides decisions. Test is evaluate-once, report-forever.

Learning Objectives

By the end of this lesson, students should be able to:

Run a test evaluation loop structurally identical to validation but on locked-out data.
Load the best validation checkpoint before testing, not the last epoch weights.
Report test metrics with dataset version, split hash, and model checkpoint ID.
Explain why repeated test evaluation inflates reported performance.
Compute task-appropriate test metrics (confusion matrix, ROC-AUC, BLEU).
Separate test code path from training scripts or guard it with a flag.

When the Test Loop Runs

Test evaluation happens after all training, validation tuning, and architecture search. In practice: train with validation monitoring → save best checkpoint → load best → run test loop once → archive results.

Split	Times evaluated during project	Allowed decisions
Train	Every batch, every epoch	Weight updates
Validation	Every epoch (or step)	Early stop, LR schedule, checkpoint
Test	Once (or rare major releases)	Final reporting only

Test Loop with Best Checkpoint

Never test with weights from the final epoch unless that epoch was also the validation best. Load state_dict from the checkpoint file written during training.

def evaluate_test(model, loader, criterion, device):
    model.eval()
    running_loss = 0.0
    all_preds, all_targets = [], []

    with torch.no_grad():
        for inputs, targets in loader:
            inputs, targets = inputs.to(device), targets.to(device)
            outputs = model(inputs)
            running_loss += criterion(outputs, targets).item()
            all_preds.append(outputs.argmax(dim=1).cpu())
            all_targets.append(targets.cpu())

    preds = torch.cat(all_preds)
    targets = torch.cat(all_targets)
    accuracy = (preds == targets).float().mean().item()
    return running_loss / len(loader), accuracy

# After training completes:
checkpoint = torch.load("checkpoints/best_val_loss.pt", map_location=device)
model.load_state_dict(checkpoint["model_state_dict"])

test_loss, test_acc = evaluate_test(model, test_loader, criterion, device)
print(f"TEST (final) | loss={test_loss:.4f} | acc={test_acc:.3f}")

Critical Mistake — “Test Every Epoch”

Logging test accuracy each epoch and publishing the maximum is optimistic bias. If you peek, you are tuning on the test set. Keep test evaluation behind if args.run_test: or a separate script invoked once.

Richer Test Metrics

Accuracy alone hides class imbalance. Collect predictions across the full loader, then compute sklearn metrics on CPU tensors.

from sklearn.metrics import classification_report, confusion_matrix

y_true = targets.numpy()
y_pred = preds.numpy()
print(classification_report(y_true, y_pred))
print(confusion_matrix(y_true, y_pred))

Documentation for Reproducibility

Log alongside test metrics: dataset split seed, preprocessing version, checkpoint path, PyTorch and CUDA versions, and commit hash. Future you (and auditors) need to reproduce the exact number.

Engineering Habit — Freeze the Test Script

Tag the git commit used for the official test run. If code changes, that is a new experiment—not an update to the old test score.

Knowledge Check

Short Answer: How many times should you tune hyperparameters using test results? Answer: Zero—test is for final evaluation only.
True/False: Test loop code differs fundamentally from validation loop code. Answer: False—same pattern; different data and invocation count.
Multiple Choice: Before testing, load: (a) last epoch weights, (b) best validation checkpoint, (c) random init. Answer: (b).
Short Answer: Why collect all predictions before computing F1? Answer: Metrics like F1 need the full set, not per-batch averages.
Short Answer: What happens if you run 50 experiments and report best test score? Answer: Optimistic bias—implicit tuning on test.
True/False: model.eval() is required for test evaluation. Answer: True.
Multiple Choice: Test set shuffle: (a) required, (b) optional, usually False, (c) forbidden by PyTorch. Answer: (b).
Short Answer: What three splits did Volume 05 establish? Answer: Train, validation, test.
Short Answer: Name two items to log with test metrics. Answer: Checkpoint path and data split seed (among others).
Multiple Choice: Validation best epoch ≠ last epoch. Test should use: (a) validation best, (b) always last, (c) epoch 1. Answer: (a).

Key Takeaways

Test loop mirrors validation but runs once on locked data after training ends.
Load the validation-best checkpoint, not arbitrary epoch weights.
Report rich metrics and full provenance; never tune on test results.
Next: Checkpoints—persist and restore training state.

Trainer’s Guide

Hands-on idea: Give students two checkpoints (best val vs last epoch). Compare test scores—often they differ meaningfully.

Discussion prompt: In a Kaggle competition, what plays the role of test vs validation?

What’s Next Learn to save the checkpoint you just loaded in Checkpoints.