ElasticNet — Module 5.4

How This Lesson Fits the Module—and Volume 05

Lasso sparsifies; Ridge stabilizes correlated weights. ElasticNet blends L1 and L2 penalties—the pragmatic default when you need some sparsity without Lasso’s instability on grouped features.

This lesson is the capstone of Volume 05: Machine Learning. You have split data, trained classical models, clustered, reduced dimensions, and now control bias–variance with regularization. Continue to Volume 06: Deep Learning where the same ideas appear as weight decay, dropout, and early stopping at scale.

Learning Objectives

By the end of this lesson, students should be able to:

Write the ElasticNet objective with mixing ratio l1_ratio.
Tune alpha and l1_ratio jointly with cross-validation.
Explain when ElasticNet outperforms pure Lasso or Ridge.
Implement ElasticNetCV in sklearn pipelines.
Map Volume 05 regularization concepts to Volume 06 deep learning practice.

ElasticNet Objective

min_w ||y − Xw||² + α [ ρ||w||₁ + (1−ρ)||w||₂² ]

sklearn ElasticNet uses l1_ratio as ρ: 1.0 = Lasso, 0.0 = Ridge, between = blend.

`l1_ratio`	Behavior	Use when
1.0	Pure Lasso	True sparsity, uncorrelated features
0.0	Pure Ridge	Dense signal, heavy collinearity
0.1–0.5 typical	Sparse + stable groups	Wide tabular with correlated blocks

sklearn Workflow

from sklearn.linear_model import ElasticNetCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("enet", ElasticNetCV(
        l1_ratio=[0.1, 0.5, 0.9],
        alphas=None,  # default path
        cv=5,
        max_iter=20000,
    )),
])
pipe.fit(X_train, y_train)
enet = pipe.named_steps["enet"]
print("alpha:", enet.alpha_, "l1_ratio:", enet.l1_ratio_)

Bridge to Volume 06

Neural networks rarely use explicit L1 on every weight, but weight decay (L2) and dropout play the same variance-reduction role. ElasticNet thinking—balance sparsity and stability—carries directly into architecture and training choices in deep learning.

Model Selection Summary

Start with Ridge when

Many small correlated effects
Need stable dense coefficients

Start with Lasso when

Clear sparsity, few active features
Inference cost dominates

Start with ElasticNet when

Wide + correlated (e.g., one-hot groups)
Lasso unstable across CV folds
Want sparsity without arbitrary single-feature picks

Critical Mistake — Tuning Only alpha, Not l1_ratio

Fixing l1_ratio=0.5 by habit hides better Ridge- or Lasso-like regions. Grid both hyperparameters (or use ElasticNetCV) on log-spaced alphas and a sensible l1_ratio grid.

Volume 05 → Volume 06 Map

Volume 05 concept	Volume 06 counterpart
Bias–variance tradeoff	Capacity vs data in network depth/width
L2 / Ridge / weight decay	AdamW, SGD weight decay
Early stopping (implicit reg.)	Validation-loss checkpoints
Dropout (preview)	Standard in CNNs, transformers
Cross-validation	Train/val splits, hyperparameter sweeps

Knowledge Check

Short Answer: What does l1_ratio=0 give? Answer: Ridge (pure L2).
True/False: ElasticNet can select groups of correlated features more stably than Lasso. Answer: True (L2 component shares shrinkage).
Multiple Choice: Tune together: (a) alpha only, (b) alpha and l1_ratio, (c) neither. Answer: (b).
Short Answer: Volume 06 analog of L2? Answer: Weight decay.
Short Answer: Why is this lesson the Vol. 05 capstone? Answer: It unifies optimization themes and bridges to deep learning.
True/False: l1_ratio=1 is pure Lasso. Answer: True.
Multiple Choice: ElasticNetCV tunes: (a) only intercept, (b) alpha and l1_ratio, (c) k. Answer: (b).
Short Answer: When prefer ElasticNet over Lasso? Answer: Correlated features where you want sparsity without arbitrary single-feature picks.
True/False: You should still scale features before ElasticNet. Answer: True.
Multiple Choice: Volume 06 begins with: (a) ANN / deep learning, (b) data collection, (c) k-means. Answer: (a).

Key Takeaways

ElasticNet = L1 + L2 blend; tune alpha and l1_ratio.
Best of both worlds on wide, correlated tabular data.
Regularization discipline completes classical ML in Volume 05.
Same principles reappear at scale in deep learning.
Volume 05 complete—continue to Volume 06: Artificial Neural Network.

Trainer’s Guide — Capstone Exercise

Hands-on idea: On a wide dataset with correlated feature blocks, students benchmark Ridge, Lasso, and ElasticNet with identical pipelines. Present best model, active feature count, and CV stability.

Discussion prompt: Draft a one-page “Volume 05 → 06” brief: which classical lessons will you revisit when training your first neural network?

Recap: ElasticNet blends L1 and L2 to stabilize sparse models on correlated features. Continue to Vol. 06 ANN.