Bias captures systematic rigidity; variance captures instability. If you retrain the same model class on different training samples, how much do predictions change? High variance means the model chases noise in each sample—the statistical root of overfitting.
Learning Objectives
By the end of this lesson, students should be able to:
- Define variance as sensitivity of predictions to the training set.
- Relate high variance to overfitting and large train–validation gaps.
- Explain the bias–variance tradeoff when increasing model complexity.
- Choose variance-reducing techniques: regularization, ensembling, more data.
- Interpret validation curves where complexity helps then hurts.
What Variance Measures
Variance is the expected squared deviation of a model’s prediction from the mean prediction across different training sets. Intuition: train two random forests on two bootstrap samples—if their decision boundaries differ wildly in sparse regions, variance is high.
| Complexity ↑ | Bias | Variance | Total error |
|---|---|---|---|
| Too simple | High | Low | High (bias-dominated) |
| Sweet spot | Moderate | Moderate | Lowest total |
| Too complex | Low | High | High (variance-dominated) |
As you increase model flexibility, bias typically falls but variance rises. Total error forms a U-shape. Model selection finds the complexity that minimizes expected error—not zero train loss.
High Variance in Practice
- Deep unpruned trees memorizing single outliers
- High-degree polynomials wiggling through every training point
- k-NN with small
k - Neural nets with many parameters vs few examples (Vol. 06)
Reducing Variance
Algorithmic
- L1/L2 regularization
- Pruning, max depth limits
- Early stopping
- Dropout (deep learning)
Data & procedure
- More training examples
- Bagging / random forests
- Cross-validation for selection
- Feature reduction (PCA)
Adding layers, trees, or polynomial degree monotonically improves training score. Without a validation metric, you cannot see variance rising. Always track out-of-sample performance.
Knowledge Check
- Short Answer: High variance corresponds to which fit problem? Answer: Overfitting.
- True/False: Increasing model complexity always reduces total error. Answer: False—variance eventually dominates.
- Multiple Choice: Random forests reduce variance primarily through: (a) boosting, (b) bagging, (c) L1 sparsity. Answer: (b).
- Short Answer: What plot shows bias–variance effects when tuning
alpha? Answer: Validation curve. - Short Answer: One way more data helps without changing the algorithm? Answer: Reduces variance of estimates.
- True/False: Deep unpruned trees tend to have high variance. Answer: True.
- Multiple Choice: Bagging reduces variance by: (a) shrinking L1, (b) averaging diverse fits, (c) adding bias on purpose only. Answer: (b).
- Short Answer: What does a large train–validation gap hint at? Answer: High variance / overfitting.
- True/False: k-NN with small k has high variance. Answer: True.
- Multiple Choice: PCA before a model can: (a) increase variance always, (b) reduce variance by dropping noisy directions, (c) create labels. Answer: (b).
Key Takeaways
- Variance = prediction instability across training samples.
- High variance ↔ overfitting; large train–val gap.
- Complexity trades lower bias for higher variance.
- Regularization and ensembling are core variance tools.
- Next: Regularization — turning theory into practice.
Hands-on idea: Students draw the classic bias–variance U-curve vs model complexity and mark where Ridge alpha sits on a real dataset.
Discussion prompt: Why might a random forest generalize better than a single deep tree on the same data?
Recap: Variance is sensitivity to the training sample—regularization and ensembles are primary controls. Continue with Regularization.