Lasso — Module 5.4

How This Lesson Fits the Module

Lasso (Least Absolute Shrinkage and Selection Operator) applies L1 regularization to linear regression. Where Ridge shrinks all coefficients, Lasso drives many to zero—delivering a sparse, deployable model from a single fit.

Learning Objectives

By the end of this lesson, students should be able to:

Fit Lasso and LassoCV in sklearn and interpret sparse coefficients.
Tune alpha to balance fit quality vs number of active features.
Read coefficient paths and stability across CV folds.
Recognize limitations with correlated features.
Position Lasso vs Ridge for production tradeoffs.

Lasso Objective

min_w ||y − Xw||₂² + α ||w||₁

Coordinate descent efficiently traces solutions along a path of alpha values. Smaller alpha → more features active; larger alpha → simpler model.

Goal	Lasso	Ridge
Feature selection	Built-in via zeros	Needs separate selection
Correlated groups	Arbitrary single pick	Shared weights
Inference speed	Fewer dot-products	All features used
Typical tabular CV	Strong when truly sparse	Strong when dense signal

sklearn Workflow

from sklearn.linear_model import LassoCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import numpy as np

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("lasso", LassoCV(cv=5, max_iter=20000, n_alphas=50)),
])
pipe.fit(X_train, y_train)

coef = pipe.named_steps["lasso"].coef_
selected = np.where(coef != 0)[0]
print(f"Active features: {len(selected)}")
print("Alpha:", pipe.named_steps["lasso"].alpha_)

Lasso for Logistic Models

LogisticRegression(penalty="l1", solver="saga") brings L1 sparsity to classification—useful for text and wide sparse matrices where linear baselines must stay interpretable.

Practical Tips

Set max_iter high enough; watch convergence warnings.
Use LassoCV or GridSearchCV on log-spaced alphas.
Inspect selected features for domain plausibility, not just CV score.
If coefficients jump when correlated cols are added, try ElasticNet.

Critical Mistake — Ignoring Convergence Warnings

Lasso on ill-scaled or poorly conditioned X may not converge. Warnings mean your sparse solution is unreliable—fix scaling, reduce collinearity, or increase max_iter before trusting zeros.

Knowledge Check

Short Answer: What does Lasso stand for? Answer: Least Absolute Shrinkage and Selection Operator.
True/False: Lasso uses L2 penalty. Answer: False—L1.
Multiple Choice: Increasing alpha in Lasso: (a) adds features, (b) removes features, (c) no effect. Answer: (b).
Short Answer: Main weakness with correlated predictors? Answer: Unstable / arbitrary feature selection.
Short Answer: sklearn class for auto-tuned Lasso? Answer: LassoCV.
True/False: LassoCV tunes alpha automatically with CV. Answer: True.
Multiple Choice: Coordinate descent is used because L1 is: (a) smooth everywhere, (b) non-smooth at zero, (c) nonconvex. Answer: (b).
Short Answer: What does a sparse coefficient vector buy at inference? Answer: Fewer features to compute/store and simpler explanations.
True/False: Very large alpha in Lasso can zero all features. Answer: True.
Multiple Choice: If Lasso picks different features each CV fold: (a) ignore it, (b) suspect correlation instability, (c) set alpha to 0. Answer: (b).

Key Takeaways

Lasso = L1 linear regression; sparse coefficients at tuned alpha.
Excellent for wide data with few true signals.
Watch convergence, scaling, and correlated-feature instability.
Pair with domain review of selected features.
Next: ElasticNet — combining L1 and L2.

Trainer’s Guide

Hands-on idea: Students compare feature count and test MSE for Lasso vs Ridge on the same pipeline. Debate which deploys faster at inference.

Discussion prompt: A Lasso model uses 12 of 4,000 features. What compliance questions should you ask before shipping?

Recap: Lasso is L1 linear regression that selects features by driving coefficients to zero. Continue with ElasticNet.