Ridge — Module 5.4

How This Lesson Fits the Module

Ridge regression is ordinary least squares plus L2 regularization. It is often the first model to try on numeric tabular problems after Linear Regression—especially when features are correlated or count is large relative to samples.

Learning Objectives

By the end of this lesson, students should be able to:

Implement Ridge with Ridge and RidgeCV in sklearn.
Tune alpha using cross-validation and validation curves.
Explain why Ridge beats OLS when p is large or features collinear.
Build pipelines with scaling + Ridge for production parity.
Extend Ridge ideas to RidgeClassifier for multiclass linear classification.

Ridge Objective

min_w ||y − Xw||₂² + α ||w||₂²

sklearn uses alpha as λ. The intercept is typically not penalized (fit_intercept=True).

`alpha`	Effect	Risk
Very small	Near OLS	High variance, unstable coefs
Moderate	Balanced shrinkage	Usually best CV performance
Very large	Weights → 0	High bias / underfitting

sklearn Workflow

from sklearn.linear_model import RidgeCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import mean_squared_error

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("ridge", RidgeCV(alphas=[0.1, 1.0, 10.0, 100.0], cv=5)),
])
pipe.fit(X_train, y_train)
pred = pipe.predict(X_test)
print("Best alpha:", pipe.named_steps["ridge"].alpha_)
print("Test MSE:", mean_squared_error(y_test, pred))

RidgeClassifier

For classification, RidgeClassifier solves a ridge-penalized least-squares formulation of labels (one-hot for multiclass). Useful as a strong linear baseline alongside Logistic Regression.

When Ridge Excels

Many features with moderate multicollinearity
n < p or near-singular design matrices
Need stable coefficients for monitoring drift
Baseline before trying non-linear models

Critical Mistake — Ridge on Unscaled Categorical One-Hots

One-hot columns and continuous features on different scales receive unequal L2 pressure. Use ColumnTransformer + scaling for numerics; consider regularization-aware encoding for high-cardinality categoricals.

Knowledge Check

Short Answer: What penalty does Ridge use? Answer: L2 (squared weights).
True/False: Larger alpha increases model flexibility. Answer: False—more shrinkage, less flexibility.
Multiple Choice: RidgeCV selects: (a) features, (b) alpha by CV, (c) learning rate. Answer: (b).
Short Answer: Why Ridge over OLS with collinear features? Answer: Lower variance, stable coefficients.
Short Answer: Is intercept penalized by default in sklearn Ridge? Answer: No.
True/False: RidgeCV can select alpha from a grid via CV. Answer: True.
Multiple Choice: StandardScaler before Ridge belongs: (a) outside any pipeline, (b) inside the same Pipeline, (c) after predict. Answer: (b).
Short Answer: What does a coefficient path vs alpha show? Answer: How weights shrink toward zero as regularization grows.
True/False: RidgeClassifier applies Ridge ideas to classification. Answer: True.
Multiple Choice: When n < p, Ridge vs OLS: (a) OLS more stable, (b) Ridge more stable, (c) identical. Answer: (b).

Key Takeaways

Ridge = linear regression + L2; tune alpha with CV.
Stabilizes solutions when features correlate or p is large.
Always scale continuous features in a pipeline.
Strong tabular baseline before trees and boosting.
Next: Lasso — L1 sparse alternative.

Trainer’s Guide

Hands-on idea: Benchmark OLS vs Ridge on a wide diabetes-style dataset with CV. Plot coefficient norms vs alpha.

Discussion prompt: When would you report Ridge coefficients to stakeholders vs switching to Lasso for sparsity?

Recap: Ridge is linear regression plus L2—scale features and tune alpha with CV. Continue with Lasso.