Overfitting — Module 5.4

How This Lesson Fits the Module

Underfitting leaves signal on the table; overfitting memorizes noise. After building expressive models in Module 5.2—trees, forests, boosting—overfitting is the primary reason offline metrics lie.

This lesson teaches you to spot the train–validation gap and motivates regularization in the rest of Module 5.4.

Learning Objectives

By the end of this lesson, students should be able to:

Define overfitting and relate it to high variance.
Interpret learning curves with low train error and high validation error.
Explain how model complexity, feature count, and data size drive overfitting.
Apply mitigations: regularization, pruning, early stopping, dropout (preview for Vol. 06).
Use cross-validation to estimate generalization honestly.

What Overfitting Means

Overfitting occurs when a model fits idiosyncrasies of the training set—including label noise and spurious correlations—that do not transfer to new data. Training metrics look excellent; validation and production metrics collapse.

Signal	Underfitting	Overfitting
Training error	High	Very low
Validation error	High	High (gap from train)
Model complexity	Too low	Too high
Bias / variance	High bias	High variance

Critical Mistake — Tuning on the Test Set

Iteratively tweaking hyperparameters until test accuracy improves is not validation—it is test-set overfitting. Hold out a true test set once, or use nested cross-validation. See Cross Validation.

Why Expressive Models Overfit

Decision trees can grow until every leaf is pure. k-NN with k=1 memorizes each point. Deep networks (Vol. 06) have millions of parameters. High capacity + limited data = memorization.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score

# Unrestricted tree — often overfits
deep = DecisionTreeClassifier(max_depth=None, min_samples_leaf=1)
print(cross_val_score(deep, X_train, y_train, cv=5).mean())

# Constrained tree — better generalization
pruned = DecisionTreeClassifier(max_depth=5, min_samples_leaf=20)
print(cross_val_score(pruned, X_train, y_train, cv=5).mean())

Detection Toolkit

Learning curves: Train error keeps falling; validation error bottoms then rises.
Validation curve: Sweep max_depth or alpha; watch validation metric peak.
Train–val gap: Large gap → overfitting suspect.
Calibration: Overconfident probabilities on held-out data.

Mitigations (Classical ML)

L1/L2 regularization (this module)
Tree depth limits, min samples per leaf
More training data or augmentation
Feature selection / PCA
Ensembling (bagging reduces variance)

Mitigations (Preview Vol. 06)

Dropout
Weight decay (L2 in SGD)
Early stopping on validation loss
Batch normalization
Data augmentation for images/text

Knowledge Check

Short Answer: Classic overfitting signature on metrics? Answer: Low train error, high validation error.
True/False: A deeper decision tree always improves production performance. Answer: False—often hurts generalization.
Multiple Choice: Overfitting is associated with: (a) high bias, (b) high variance, (c) low variance. Answer: (b).
Short Answer: Why is k-NN with k=1 prone to overfitting? Answer: It memorizes single training points.
Short Answer: Safe way to compare hyperparameters? Answer: Cross-validation on training data, not repeated test peeking.
True/False: Early stopping can reduce overfitting. Answer: True.
Multiple Choice: More training data usually: (a) increases overfitting, (b) reduces overfitting/variance, (c) has no effect. Answer: (b).
Short Answer: What is memorization here? Answer: Fitting noise or idiosyncrasies of the training sample.
True/False: Dropout is a deep-learning regularizer against overfitting. Answer: True.
Multiple Choice: Validation score peaking then falling as depth grows shows: (a) underfitting throughout, (b) overfitting at high complexity, (c) leakage only. Answer: (b).

Key Takeaways

Overfitting = great on train, poor on unseen data.
Expressive models and small datasets are a risky combination.
Regularization and validation discipline are your primary defenses.
Overfitting maps to high variance in the bias–variance framework.
Next: Bias — formalizing the tradeoff.

Trainer’s Guide

Hands-on idea: Plot validation curve for DecisionTreeClassifier.max_depth from 1 to 30. Students mark the sweet spot where CV score peaks.

Discussion prompt: 99% train accuracy, 62% test—list three non-leakage explanations and fixes.

Recap: Overfitting is low train error with high validation error—constrain capacity and validate honestly. Continue with Bias.