Underfitting — Module 5.4

How This Lesson Fits the Module

After clustering and dimensionality reduction in Module 5.3, you have trained models that can memorize or miss patterns. Underfitting is the failure mode where a model is too simple to capture the signal in your data—high error on training and test sets.

Recognizing underfitting is the first step in the bias–variance tradeoff. This lesson sets the vocabulary for everything that follows in Model Optimization.

Learning Objectives

By the end of this lesson, students should be able to:

Define underfitting and distinguish it from overfitting and good fit.
Read learning curves that show high training and validation error.
Identify common causes: insufficient model capacity, weak features, or excessive regularization.
Apply remedies: richer features, more complex models, longer training, or lower regularization.
Connect underfitting to high bias in the bias–variance decomposition.

What Underfitting Means

An underfit model fails to learn the underlying relationship between features and labels. It performs poorly on training data—so the problem is not generalization; the model never learned the task in the first place.

Fit quality	Training error	Validation error	Typical cause
Underfit	High	High (similar to train)	Model too simple
Good fit	Moderate/low	Close to train	Balanced capacity
Overfit	Very low	Much higher than train	Model too complex

Intuition — Straight Line Through a Curve

Fitting a degree-1 polynomial to clearly non-linear data will underfit: both train and test MSE stay high. Adding polynomial terms or switching to a tree-based model increases capacity until validation error improves.

Diagnosing Underfitting

Plot learning curves: training and validation metric vs. training set size or training epochs. Underfitting shows both curves high and converging together with a large gap to the best achievable error.

from sklearn.model_selection import learning_curve
from sklearn.linear_model import LinearRegression
import numpy as np

train_sizes, train_scores, val_scores = learning_curve(
    LinearRegression(), X, y, cv=5, scoring="neg_mean_squared_error"
)
# Both curves near zero (neg MSE) and flat → likely underfit on non-linear data

Common Causes

Model-side

Linear model on non-linear data
Shallow tree (max_depth=1)
Too few boosting rounds
Heavy regularization (large λ)

Data-side

Missing informative features
Features not scaled when algorithm requires it
Label noise dominating signal
Insufficient training samples for complexity attempted

Critical Mistake — Fixing Underfitting with More Data

Collecting more rows rarely fixes underfitting caused by wrong model class. If train error is already high, more data usually keeps both errors high. Increase capacity or improve features first; then revisit sample size.

Remedies

Strategy	Example
Increase model complexity	Deeper trees, more hidden units (later in Vol. 06), interaction terms
Reduce regularization	Lower `alpha` in Ridge/Lasso
Engineer features	Polynomials, domain aggregates, embeddings
Train longer	More epochs with early stopping on validation loss
Lower decision threshold bar	Relax constraints (e.g., SVM `C` larger)

Knowledge Check

Short Answer: How do train and val error compare when underfitting? Answer: Both are high and similar.
True/False: Underfitting always means you need more training data. Answer: False—often you need more model capacity or better features.
Multiple Choice: High bias corresponds to: (a) underfitting, (b) overfitting, (c) neither. Answer: (a).
Short Answer: One sklearn sign of underfitting on non-linear data with LinearRegression? Answer: High MSE on both train and test.
Short Answer: First diagnostic plot to run? Answer: Learning curve (train vs validation error).
True/False: Heavy regularization can cause underfitting. Answer: True.
Multiple Choice: max_depth=1 on a complex target: (a) overfits, (b) underfits, (c) is unbiased. Answer: (b).
Short Answer: Name one remedy besides more rows. Answer: Richer features, more capacity, or lower regularization.
True/False: Polynomial features can reduce bias on nonlinear data. Answer: True.
Multiple Choice: Train and val error both near chance: (a) overfit, (b) underfit or bad features/labels, (c) perfect. Answer: (b).

Key Takeaways

Underfitting = high error on training data; the model is too weak.
Train and validation errors are both poor and close together.
Fix with capacity, features, or less regularization—not blindly with more rows.
Underfitting is the high-bias side of the tradeoff.
Next: Overfitting — the opposite failure mode.

Trainer’s Guide

Hands-on idea: Students fit LinearRegression vs PolynomialFeatures(degree=3) on a synthetic sine dataset. Compare train/test MSE and sketch learning curves.

Discussion prompt: Your stakeholder says accuracy is 55% on train and 54% on test. Underfit, overfit, or inconclusive? What would you try first?

Recap: Underfitting is high error on train and validation—the model is too simple or poorly featured. Continue with Overfitting.