← Master Index
Vol. 05 Module 5.4 Lecture

L2

Model Optimization

How This Lesson Fits the Module

Where L1 encourages sparsity, L2 penalizes the sum of squared weights: λ Σwᵢ². Coefficients shrink toward zero but rarely become exactly zero. L2 is the backbone of Ridge regression and appears as weight decay in deep learning (Volume 06).

Learning Objectives

By the end of this lesson, students should be able to:

  • Write the L2 penalty and relate it to Gaussian prior on weights (conceptual).
  • Explain why L2 distributes weight among correlated features.
  • Derive that Ridge has a closed-form solution (normal equations + λI).
  • Contrast L2 smooth shrinkage vs L1 corner sparsity.
  • Prepare for Ridge regression implementation in sklearn.

The L2 Penalty

minw ||y − Xw||² + λ ||w||2²

Geometrically, the L2 constraint region is a sphere. Loss contours typically intersect the sphere edge, not axes—so weights shrink but stay non-zero.

AspectL2 behavior
Correlated featuresWeights shared across group
MulticollinearityStabilizes (XTX + λI) invertible
SparsityGenerally no exact zeros
OptimizationSmooth; closed form for Ridge
Multicollinearity Fix

When two features are nearly identical, ordinary least squares blows up coefficient variance. Adding λ to the diagonal of XTX tames ill-conditioning—a key reason Ridge is the default linear baseline on wide tabular data.

Closed-Form Ridge Solution

# Ridge solution: w = (X^T X + λI)^(-1) X^T y from sklearn.linear_model import Ridge from sklearn.preprocessing import StandardScaler from sklearn.pipeline import Pipeline model = Pipeline([ ("scale", StandardScaler()), ("ridge", Ridge(alpha=1.0)), ]) model.fit(X_train, y_train)

L1 vs L2 Decision Guide

Prefer L2 when

  • Many correlated predictors
  • All features may contribute slightly
  • Stable coefficients matter
  • You need a smooth, dense model

Prefer L1 when

  • True sparsity expected
  • Feature selection for deployment
  • Interpretability via few active features
  • High p, suspected many irrelevant cols
Critical Mistake — Comparing Unscaled Coefficients

L2 shrinks large-magnitude features more in raw units. Interpret coefficients only after scaling, or use standardized coefficients from a pipeline.

Knowledge Check

  1. Short Answer: L2 penalty formula? Answer: λ times sum of squared weights.
  2. True/False: L2 typically produces sparse models with many zero weights. Answer: False.
  3. Multiple Choice: L2 helps multicollinearity by: (a) dropping columns, (b) adding λ to diagonal, (c) increasing tree depth. Answer: (b).
  4. Short Answer: Deep learning name for L2 on weights? Answer: Weight decay.
  5. Short Answer: When correlated features exist, L1 vs L2 selection behavior? Answer: L1 picks one; L2 shares weight.
  6. True/False: The L2 constraint is a disk/sphere (smooth). Answer: True.
  7. Multiple Choice: Closed-form Ridge adds λ to: (a) labels, (b) XTX diagonal, (c) k. Answer: (b).
  8. Short Answer: Why does L2 rarely zero coefficients? Answer: Squared penalty shrinks smoothly without a corner at zero.
  9. True/False: Strong L2 can still underfit. Answer: True.
  10. Multiple Choice: sklearn class for L2 linear regression: (a) Lasso, (b) Ridge, (c) DBSCAN. Answer: (b).

Key Takeaways

  • L2 = squared coefficient penalty; smooth shrinkage.
  • Handles correlated features better than L1.
  • Ridge has closed-form solution; stable with multicollinearity.
  • Reappears as weight decay in neural networks (Vol. 06).
  • Next: Ridge — L2 linear regression in practice.
Trainer’s Guide

Hands-on idea: Create two perfectly correlated features. Compare OLS, Lasso, and Ridge coefficients side by side.

Discussion prompt: Why might production teams prefer dense Ridge over sparse Lasso for monitoring?

Recap: L2 squares weights for smooth shrinkage and stable coefficients under collinearity. Continue with Ridge.