Overfitting and Underfitting describe behavior; bias explains why in the language of statistical learning theory. Bias measures how wrong your model’s assumptions are on average across training sets drawn from the same distribution.
Pair this lesson with Variance to complete the bias–variance decomposition that guides all of Model Optimization.
Learning Objectives
By the end of this lesson, students should be able to:
- Define bias (estimation bias) in the ML sense—distinct from fairness bias.
- Relate high bias to underfitting and systematic prediction error.
- State the bias–variance decomposition of expected prediction error.
- Recognize model choices that increase or decrease bias.
- Distinguish statistical bias from social/fairness bias in documentation.
Bias in the Statistical Sense
Bias is the error from erroneous assumptions in the learning algorithm. A linear model trying to fit a quadratic truth has high bias: it systematically under-predicts in some regions and over-predicts in others, even with infinite data.
Expected test error decomposes (for squared loss) as:
Error = Bias² + Variance + Irreducible Noise
| Term | Meaning | Typical symptom |
|---|---|---|
| Bias² | Systematic deviation from true function | High train error |
| Variance | Sensitivity to training sample | Train–val gap |
| Noise | Randomness in labels | Floor on achievable error |
In this module, bias means estimation bias (underfitting). In responsible AI work, bias also means unfair treatment of groups. Both matter in production; keep definitions explicit in reports and model cards.
High Bias = Underfitting
Models with high bias are too rigid: linear decision boundary for XOR, shallow stump for complex tabular data, heavy L2 pushing all weights toward zero. They cannot represent the true relationship regardless of how much you train.
Lower Bias (more flexible)
- Polynomial / spline features
- Deep neural networks
- Unpruned trees, boosting
- Small regularization λ
Higher Bias (more rigid)
- Linear/logistic regression (linear boundary)
- Naive Bayes independence assumption
- Strong regularization
- Very shallow trees
Bias in Practice
Dropout, bagging, and stronger regularization reduce variance. Applied to an already underfit model, they make training error worse. Diagnose with learning curves before choosing a remedy.
Knowledge Check
- Short Answer: What does high bias mean for train error? Answer: It tends to be high (underfitting).
- True/False: Bias in this lesson is the same as demographic fairness bias. Answer: False—same word, different concepts.
- Multiple Choice: Bias² + variance + noise equals: (a) train loss, (b) expected prediction error, (c) learning rate. Answer: (b).
- Short Answer: One modeling choice that increases bias? Answer: Strong regularization or overly simple model class.
- Short Answer: Can you eliminate irreducible noise? Answer: No—it sets a performance floor.
- True/False: Linear models have relatively high bias on highly nonlinear problems. Answer: True.
- Multiple Choice: Lowering bias usually: (a) always lowers variance too, (b) often raises variance, (c) removes noise. Answer: (b).
- Short Answer: What is irreducible error? Answer: Noise in the data-generating process that no model can remove.
- True/False: Naive Bayes’ independence assumption is a source of bias. Answer: True.
- Multiple Choice: High-bias models typically show: (a) tiny train error, (b) high train error, (c) huge train–val gap only. Answer: (b).
Key Takeaways
- Bias measures systematic error from wrong or too-simple assumptions.
- High bias ↔ underfitting; high train and test error.
- Error decomposes into bias², variance, and noise.
- Clarify statistical vs fairness “bias” in documentation.
- Next: Variance — the overfitting counterpart.
Hands-on idea: Bootstrap 50 fits of linear vs polynomial regression on the same synthetic data. Plot prediction spread (variance) vs average distance to truth (bias).
Discussion prompt: When stakeholders conflate fairness bias and model bias, how do you reframe the conversation?
Recap: Bias is systematic error from overly simple assumptions—the underfitting side of the tradeoff. Continue with Variance.