L1 — Module 5.4 | Codextroop

How This Lesson Fits the Module

Regularization adds penalties to the loss. L1 (Lasso penalty) uses the sum of absolute values of weights: λ Σ|wᵢ|. Unlike L2, L1 can drive coefficients exactly to zero—performing built-in feature selection.

Learning Objectives

By the end of this lesson, students should be able to:

Write the L1 penalty and explain geometric sparsity (diamond constraint).
Contrast L1 with L2 shrinkage behavior on correlated features.
Interpret sparse coefficient vectors for interpretability and deployment.
Recognize when L1 helps: high-dimensional, many irrelevant features.
Connect L1 to Lasso regression in sklearn.

The L1 Penalty

For linear models, the L1-regularized objective is:

min_w ||y − Xw||² + λ ||w||₁

The L1 ball has corners on the axes. When the loss contour touches a corner, some wᵢ become exactly zero.

Property	L1	L2 (preview)
Penalty	Sum of \|wᵢ\|	Sum of wᵢ²
Sparsity	Yes—exact zeros	Rarely exact zeros
Correlated features	Picks one, zeros others	Shares weight
Differentiable at 0	No (subgradient methods)	Yes

When Sparsity Wins

Genomics, text bag-of-words, and wide ad-tech logs often have thousands of features but few truly predictive ones. L1 surfaces a compact subset for faster inference and clearer audits.

sklearn Preview

from sklearn.linear_model import Lasso
import numpy as np

lasso = Lasso(alpha=0.05, max_iter=10000)
lasso.fit(X_train, y_train)

nonzero = np.sum(lasso.coef_ != 0)
print(f"Features kept: {nonzero} / {len(lasso.coef_)}")
print(lasso.coef_[:10])  # many zeros at sufficient alpha

L1 Caveats

Correlated features: L1 may arbitrarily select one from a correlated group—unstable for interpretation.
Scale sensitivity: Requires standardized features in practice.
Convex but non-smooth: Coordinate descent (used in sklearn) handles this efficiently.

Critical Mistake — Treating L1 Zeros as Causal Truth

Zero weight means “not used at this λ,” not “feature is irrelevant in nature.” Stability selection or ElasticNet may be needed when features are correlated.

Knowledge Check

Short Answer: What norm defines L1? Answer: ||w||₁ = sum of absolute weights.
True/False: L1 always shrinks weights but never to exactly zero. Answer: False—it can zero coefficients.
Multiple Choice: L1 is best known for: (a) sparsity, (b) analytic closed form, (c) no hyperparameters. Answer: (a).
Short Answer: Why standardize before L1? Answer: Penalty must be comparable across feature scales.
Short Answer: sklearn class implementing L1 linear regression? Answer: Lasso.
True/False: The L1 constraint set is a diamond (cross-polytope). Answer: True.
Multiple Choice: L1 feature selection is: (a) embedded in the optimizer, (b) always wrapper RFE, (c) unsupervised. Answer: (a).
Short Answer: What happens to a coefficient that hits zero under L1? Answer: That feature is unused by the linear model.
True/False: L1 is non-differentiable at zero. Answer: True.
Multiple Choice: For correlated groups, prefer: (a) pure L1 always, (b) ElasticNet or L2 sharing, (c) no penalty. Answer: (b).

Key Takeaways

L1 = sum of absolute coefficients; encourages sparsity.
Exact zeros enable feature selection and simpler models.
Unstable with correlated features; consider ElasticNet.
Scale features; tune alpha with CV.
Next: L2 — smooth shrinkage without sparsity.

Trainer’s Guide

Hands-on idea: Plot coefficient paths as alpha increases—watch coefficients hit zero at different times.

Discussion prompt: For compliance, is a sparse Lasso model always more explainable than a dense Ridge model?

Recap: L1 penalizes absolute weights and drives sparsity—useful for feature selection after scaling. Continue with L2.