Validation Set — Module 5.1

How This Lesson Fits the Module

Training teaches; testing certifies. The validation set sits between them: data you use repeatedly to compare models, pick features, and tune hyperparameters without burning the final test set.

Think of validation as the practice exam you are allowed to retake—as long as you do not confuse it with the sealed final.

Learning Objectives

By the end of this lesson, students should be able to:

Define the validation set and distinguish it from train and test.
Use validation metrics for model and feature selection.
Create train/validation splits with train_test_split.
Explain validation-set overfitting from too many experiments.
Map the classic 60/20/20 or 80/10/10 split strategies to team workflow.
Prefer cross-validation when validation data is scarce (next lesson).

Three-Way Split Workflow

Many teams hold out test first, then split remaining rows into train and validation. Validation scores guide daily decisions; test scores appear in release reviews.

Split	Typical share	Decisions enabled
Train	60–80%	Fit models and preprocessing
Validation	10–20%	Pick algorithm, features, thresholds
Test	10–20%	Final sign-off only

Creating Train and Validation Sets

Hold out test once, then split the development pool. Use stratify=y for classification when classes are imbalanced.

from sklearn.model_selection import train_test_split

X_dev, X_test, y_dev, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

X_train, X_val, y_train, y_val = train_test_split(
    X_dev, y_dev, test_size=0.25, stratify=y_dev, random_state=42
)
# 0.25 of 0.8 = 0.2 → ~60/20/20 overall

clf.fit(X_train, y_train)
val_auc = roc_auc_score(y_val, clf.predict_proba(X_val)[:, 1])

Critical Mistake — Validation Worship

Running fifty model variants and picking the best validation score overfits the validation set. Use nested CV, a fresh validation slice, or register that the winning model was chosen after many trials (metrics are optimistic).

What You Do on Validation

Compare algorithms, prune features, set classification thresholds, and early-stop iterative training. Log every validation experiment to avoid hidden cherry-picking.

Good validation use

Compare logistic vs random forest
Pick probability threshold for F1 target
Detect obvious overfitting train vs val

Save for test only

Executive go/no-go number
Benchmark after feature freeze
Regulatory submission metric

Engineering Habit — Experiment Log

Log validation metric, git SHA, data snapshot, and hyperparameters for every run. When validation improves mysteriously, the log shows whether data or code changed.

Knowledge Check

Short Answer: What is the validation set for? Answer: Iterative model comparison and tuning without using the test set.
True/False: Validation and test sets can be swapped freely. Answer: False—they have different roles.
Multiple Choice: After 100 validation-driven trials, validation metric is: (a) unbiased, (b) optimistically biased, (c) identical to test. Answer: (b).
Short Answer: Why hold out test before train/val split? Answer: Prevents test rows from leaking into any tuning data.
Short Answer: What does stratify=y preserve? Answer: Class proportions across splits.
True/False: Validation labels are used inside fit. Answer: False—only for evaluation after predict.
Multiple Choice: 60/20/20 split means test is: (a) 20%, (b) 60%, (c) 40%. Answer: (a).
Short Answer: When is a single validation set weak? Answer: Small data or high variance metrics—use CV.
Short Answer: Name one decision made on validation. Answer: e.g., choose max_depth or probability threshold.
Multiple Choice: random_state=42 mainly ensures: (a) reproducible splits, (b) better accuracy, (c) no need for test set. Answer: (a).

Key Takeaways

Validation enables safe iteration; test remains locked for final reporting.
Hold out test first, then split train/validation from the dev pool.
Many experiments on one validation slice inflate scores—log trials.
Next: Train-Test Split—mechanics and options in sklearn.

Trainer’s Guide

Hands-on idea: Students implement 60/20/20 with two nested train_test_split calls and verify proportions with value_counts on each y split.

Discussion prompt: Your validation AUC improves for five weeks but test is flat. What hypotheses do you investigate?

Recap: The validation set steers hyperparameter choices without touching the final test. Continue with Train-Test Split.