Cross Validation — Module 5.1

How This Lesson Fits the Module

A single train/validation split can be lucky or unlucky. Cross-validation (CV) rotates multiple train/validation partitions so metrics are averaged across folds—more stable decisions with limited labeled data.

AI engineers use CV daily inside hyperparameter search and model comparison, always with pipelines so preprocessing refits per fold (Volume 04 leakage lesson).

Learning Objectives

By the end of this lesson, students should be able to:

Explain k-fold cross-validation and why it reduces split variance.
Run cross_val_score and interpret mean and std of metrics.
Choose among KFold, StratifiedKFold, GroupKFold, and TimeSeriesSplit.
Nested CV conceptually for unbiased hyperparameter evaluation.
Wrap estimators in Pipeline before CV.
Report CV results with fold count and scoring metric in experiment logs.

How K-Fold CV Works

Data are divided into k folds. Each round trains on k-1 folds and validates on the held-out fold. The metric is averaged across rounds. Every row serves as validation exactly once.

CV variant	Use when
`KFold`	Regression or balanced classification
`StratifiedKFold`	Imbalanced classification
`GroupKFold`	Rows grouped by user/document
`TimeSeriesSplit`	Ordered time; train always before test

cross_val_score with Pipelines

Never CV on manually preprocessed data. Put preprocessing and model in one Pipeline so each fold fits transforms on training folds only.

from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("clf", LogisticRegression(max_iter=1000)),
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(pipe, X, y, cv=cv, scoring="roc_auc")

print(f"ROC-AUC: {scores.mean():.3f} (+/- {scores.std():.3f})")

Critical Mistake — CV on Leaked Features

Running CV after target encoding on the full dataset still leaks. Every transformer must live inside the pipeline passed to cross_val_score or GridSearchCV.

Group and Time Series CV

from sklearn.model_selection import GroupKFold, TimeSeriesSplit

gkf = GroupKFold(n_splits=5)
group_scores = cross_val_score(pipe, X, y, cv=gkf.split(X, y, groups=df["user_id"]))

tscv = TimeSeriesSplit(n_splits=5)
ts_scores = cross_val_score(pipe, X, y, cv=tscv, scoring="neg_mean_absolute_error")

Engineering Habit — Report Fold Variance

Publish mean and standard deviation. AUC 0.82 ± 0.01 is stable; 0.82 ± 0.08 warrants more data or simpler models before shipping.

Nested Cross-Validation (Concept)

Outer CV estimates generalization; inner CV tunes hyperparameters. Without nesting, tuning on the same folds you report inflates scores. GridSearchCV with a held-out test set is the practical alternative covered next lesson.

Knowledge Check

Short Answer: What does 5-fold CV do? Answer: Trains and validates five times, each fold held out once.
True/False: CV replaces the need for a final test set. Answer: False—still hold out locked test data.
Multiple Choice: Imbalanced classification CV: (a) KFold, (b) StratifiedKFold, (c) no CV. Answer: (b).
Short Answer: Why pipeline inside CV? Answer: Preprocessing refits on training folds only per round.
Short Answer: What does high std across folds suggest? Answer: Unstable model or insufficient/split-sensitive data.
True/False: TimeSeriesSplit shuffles rows randomly. Answer: False—expanding train window forward in time.
Multiple Choice: GroupKFold prevents: (a) group leakage, (b) GPU OOM, (c) missing values. Answer: (a).
Short Answer: What is nested CV? Answer: Outer loop for evaluation, inner loop for tuning without contaminating outer folds.
Short Answer: scoring="roc_auc" in cross_val_score sets what? Answer: The metric computed on each validation fold.
Multiple Choice: CV on 500 rows with 50 features — concern: (a) high variance, (b) no concern, (c) labels unnecessary. Answer: (a) — consider regularization or more data.

Key Takeaways

CV averages performance across folds for stabler estimates than one split.
Pick fold strategy to match data structure (stratify, group, time).
Always CV the full pipeline, not pre-leaked matrices.
Next: Hyperparameter—tuning with CV-backed search.

Trainer’s Guide

Hands-on idea: Compare single-split AUC vs 5-fold mean/std on the same pipeline. Discuss which decision (ship vs iterate) each would support.

Discussion prompt: When is CV too expensive for a nightly retrain job? What approximations are acceptable?

Recap: Cross-validation rotates folds so every training row helps estimate generalization. Continue with Hyperparameter.