Mode told you which value appears most often—the center of categorical mass. Mean and median locate the center of numeric data. Variance answers the next question: how spread out are the values around that center?
In machine learning, spread is not a footnote. Features with large variance dominate distance-based models and gradient steps unless scaled. Loss functions often are variances (mean squared error). Regularization penalizes weight variance across layers. Batch normalization tracks running variance to stabilize training. Understanding Var(X) is prerequisite for standardization, confidence in predictions, and reading learning curves.
Variance measures average squared deviation from the mean. It is the workhorse of spread—standard deviation is simply its square root.
Learning Objectives
By the end of this lesson, students should be able to:
- Define population variance Var(X) and compute it from a finite population.
- Define sample variance s2 and explain why the denominator is n − 1 instead of n.
- Interpret variance as expected squared deviation: Var(X) = E[(X − μ)2].
- Distinguish when to use population vs sample formulas in ML contexts (full data vs held-out estimate).
- State the computational (sum-of-squares) formula and apply it to small datasets by hand.
- Explain Bessel’s correction at a high level as unbiasedness of sample variance.
- Connect variance to MSE loss, feature scaling motivation, and batch normalization statistics.
- Recognize that variance is in squared units and why we often report standard deviation instead.
Introduction: Measuring Spread
Two datasets can share the same mean yet look completely different. Consider daily inference latency (milliseconds):
- Service A: 48, 49, 50, 51, 52 — mean 50, tightly clustered
- Service B: 10, 30, 50, 70, 90 — mean 50, widely scattered
Mean alone cannot distinguish them. Variance quantifies how far observations typically fall from the mean—large variance means high dispersion, small variance means consistency.
For AI practitioners, variance appears in three layers: data (feature spread), models (weight and activation spread), and objectives (squared-error losses). This lecture builds the statistical definition; the next lecture (Standard Deviation) translates it into original units and z-scores.
Population Variance
For a random variable X with mean μ = E[X], the population variance is:
Var(X) = E[(X − μ)2]
For a finite population of N values x1, …, xN with population mean μ, the population variance (sometimes denoted σ2) is:
σ2 = (1/N) ∑i=1N (xi − μ)2
Read the definition aloud: “variance equals the average squared distance from the mean.” Squaring ensures positive contributions from both above-mean and below-mean deviations, and penalizes large outliers more than small ones.
| Symbol | Name | Meaning |
|---|---|---|
| Var(X) or σ2 | Population variance | Squared spread around the true/population mean μ |
| μ | Population mean | E[X] or average of the full population |
| xi − μ | Deviation | How far observation i is from center |
| (xi − μ)2 | Squared deviation | Always nonnegative; amplifies large deviations |
Data: 48, 49, 50, 51, 52. Mean μ = 50.
- Deviations: −2, −1, 0, 1, 2
- Squared: 4, 1, 0, 1, 4 — sum = 10
- Population variance: σ2 = 10/5 = 2 (ms2)
Data: 10, 30, 50, 70, 90. Mean μ = 50.
- Squared deviations sum to 4000 + 400 + 0 + 400 + 1600 = 6400
- σ2 = 6400/5 = 1280 (ms2)
Same mean, radically different variance. Service B is unreliable for latency-sensitive inference.
Computational Formula
Expanding the definition yields an equivalent formula useful for hand calculation and one-pass algorithms:
Var(X) = E[X2] − (E[X])2
For a finite population:
σ2 = (1/N) ∑ xi2 − μ2
In code, prefer numerically stable library functions over naive two-pass formulas on large arrays:
import numpy as np
latencies = np.array([48, 49, 50, 51, 52])
np.var(latencies, ddof=0) # population variance (divide by N)
np.var(latencies, ddof=1) # sample variance (divide by N-1)Sample Variance: Estimating Spread from Data
In ML you almost never possess the full population—you have a sample (training set, batch, logged events). You estimate population variance with sample variance s2:
s2 = (1/(n − 1)) ∑i=1n (xi − x̄)2
where n is sample size and x̄ is the sample mean. The denominator n − 1 is Bessel’s correction.
Population Variance
Divide by N. Use when you have the entire population or define variance of a theoretical distribution.
NumPy: np.var(x, ddof=0)
Sample Variance
Divide by n − 1. Use when estimating spread from a subset to generalize beyond it.
NumPy: np.var(x, ddof=1) (default in pandas .var())
ML Practice
Feature scaling (StandardScaler) uses sample std with n − 1 by default. MSE on a batch divides by n for the loss scalar—a mean, not an unbiased variance estimate.
Rule: Match the convention of your library; document ddof in reproducibility notes.
Bessel’s Correction (Brief)
Using the sample mean x̄ instead of the unknown true mean μ tightens deviations—they are forced to sum to zero. Dividing by n would systematically underestimate population variance. Dividing by n − 1 compensates, making s2 an unbiased estimator of σ2:
E[s2] = σ2
With n sample points, you estimate one parameter (the mean) before measuring spread. Only n − 1 deviations are “free” once the mean is fixed—the last one is determined by the others summing to zero. The correction accounts for that lost degree of freedom. For large n (typical in deep learning batches), n vs n − 1 is negligible; for tiny samples it matters.
You do not need to derive Bessel’s correction to build models—but you should know ddof=1 vs ddof=0 when comparing numbers across pandas, NumPy, and spreadsheets.
Variance and Spread in Machine Learning
Mean Squared Error as Variance Around Predictions
Regression loss MSE = (1/n) ∑(yi − ŷi)2 is the mean squared deviation of predictions from targets—structurally identical to variance, but centered on predictions rather than the sample mean of y. Minimizing MSE on training data seeks low spread of residuals.
Feature Variance and Dominance
Consider two features: age (variance ≈ 100) and income (variance ≈ 1012). Unscaled distance metrics (k-NN, RBF kernels) and gradient descent on raw inputs are dominated by high-variance columns. Computing per-feature variance in EDA flags columns needing standardization.
Batch Normalization
BatchNorm layers normalize activations using batch mean and variance (with learned scale/shift and running statistics at inference). Stabilizing activation variance reduces internal covariate shift and allows higher learning rates—variance is not just descriptive, it is a training intervention.
df.var() per numeric column; flag near-zero variance (constant features)
↓
Preprocessing — StandardScaler stores mean and variance from training set
↓
Training — MSE / Var-style losses; BatchNorm tracks running variance
↓
Monitoring — Residual variance and prediction spread on validation drift checks
Variance of Sums (Preview)
For independent random variables, variances add: Var(X + Y) = Var(X) + Var(Y). This underlies why averaging n independent noisy measurements reduces variance by factor n—relevant for ensemble methods and why more data stabilizes estimates.
Properties and Units
- Non-negativity: Var(X) ≥ 0; equals 0 only when X is constant.
- Scaling: Var(aX + b) = a2 Var(X) for constant a, b. Adding a constant does not change variance.
- Units: If X is in milliseconds, Var(X) is in ms2—hard to interpret. Standard deviation σ = √Var(X) restores original units (covered next lecture).
- Zero variance features: Columns with Var = 0 carry no information; drop them before training.
Common Misconceptions
Why people believe it: They are taught as separate formulas.
Reality: Standard deviation is √Var(X). Same information, different units.
Why people believe it: The population formula uses N in the denominator.
Reality: Unbiased estimation of population variance from a sample uses n − 1. Libraries differ by default—check ddof.
Why people believe it: Variance sounds like “variation carries signal.”
Reality: Variance measures spread, not predictive power. A feature can have huge variance and zero correlation with the target (e.g., random IDs). Use mutual information or model-based importance for informativeness.
Why people believe it: Both average squared deviations.
Reality: Sample variance centers on the sample mean of one variable; MSE centers on model predictions vs targets. Denominators and purposes differ (estimation vs optimization).
Quick Knowledge Check
- Short Answer: Write the population variance formula for values x1, …, xN with mean μ. Answer: σ2 = (1/N) ∑(xi − μ)2.
- True/False: Variance can be negative. Answer: False—squared deviations are nonnegative.
- Short Answer: What is Bessel’s correction? Answer: Using n − 1 instead of n in the sample variance denominator for unbiased estimation of population variance.
- Multiple Choice: In NumPy, which gives sample variance? (a)
np.var(x, ddof=0), (b)np.var(x, ddof=1), (c)np.std(x)only, (d) both a and b. Answer: (b). - Computation: Values 2, 4, 4, 4, 6 have mean 4. What is the population variance? Answer: Deviations −2,0,0,0,2; squared sum = 8; σ2 = 8/5 = 1.6.
- Short Answer: Why is variance in squared units? Answer: Because deviations are squared in the definition.
- True/False: Adding 100 to every value changes the variance. Answer: False—Var(X + b) = Var(X) for constant b.
- Short Answer: Why drop near-zero-variance features before training? Answer: They are (near) constant and provide no discriminative signal; can cause numerical issues.
- Multiple Choice: MSE in regression most closely resembles: (a) mean absolute deviation, (b) mean squared deviation from predictions, (c) mode, (d) median. Answer: (b).
- Short Answer: For large batch sizes, does n vs n−1 matter much in practice? Answer: No—the ratio approaches 1; difference is negligible for large n.
Key Takeaways
- Variance is average squared deviation from the mean: Var(X) = E[(X − μ)2].
- Population variance divides by N; sample variance divides by n − 1 (Bessel’s correction).
- Large variance means wide spread; zero variance means a constant feature.
- MSE loss shares the squared-deviation structure but measures prediction error, not feature spread.
- Feature variance drives scale sensitivity in ML—motivation for standardization in the next lecture.
- Batch normalization and ensemble averaging leverage variance algebra to stabilize training.
- Always check
ddofwhen comparing variance across tools; pandas defaults differ from NumPy.
Further Reading & References
Books
- Introduction to Statistical Learning — James, Witten, Hastie, Tibshirani. Chapter 2: bias-variance tradeoff (conceptual link to model variance).
- Deep Learning — Goodfellow, Bengio, Courville. Normalization and optimization chapters.
Documentation
- NumPy var —
ddofparameter explained - sklearn StandardScaler — Uses variance for scaling
Hands-on: Give students two CSV columns with identical means. Have them compute df.var() and predict which feature will dominate k-NN before and after StandardScaler.
ddof drill: On a sample of 5 points, compute variance by hand with n and n−1. Compare to np.var(x, ddof=0) and ddof=1.
Bridge: Variance is in squared units—the next lecture introduces standard deviation σ, z-scores, and the empirical rule for interpreting spread in original units.