Linear Regression — Module 5.2

How This Lesson Fits the Module

Module 5.1 taught splits, metrics, and pipelines. Linear regression is the first concrete algorithm: a supervised model that predicts a continuous target as a weighted sum of features. Every later classifier and tree ensemble builds on the ideas of loss, coefficients, and generalization introduced here.

Start with the simplest model that could work—then justify complexity only when linear assumptions break down.

Learning Objectives

By the end of this lesson, students should be able to:

State the linear regression hypothesis and mean squared error (MSE) objective.
Fit and interpret LinearRegression and regularized variants (Ridge, Lasso).
Evaluate regression with r2_score, MAE, and RMSE on a held-out test set.
Recognize when scaling and feature engineering matter for linear models.
Diagnose underfitting (high bias) and overfitting (high variance) from learning curves.

What Linear Regression Does

Linear regression models a continuous target y as a linear function of features x: ŷ = w₀ + w₁x₁ + … + wₙxₙ. Training finds weights that minimize squared error between predictions and true values. Despite its simplicity, it remains a strong baseline for pricing, demand forecasting, and any tabular problem with roughly linear relationships.

Concept	Meaning	sklearn
Ordinary least squares (OLS)	Closed-form or iterative solution minimizing MSE	`LinearRegression()`
Ridge (L2)	Shrinks large coefficients; handles multicollinearity	`Ridge(alpha=1.0)`
Lasso (L1)	Can zero out weak features; sparse models	`Lasso(alpha=0.1)`
Intercept	Baseline prediction when all features are zero	`fit_intercept=True`

End-to-End sklearn Example

Use the California housing dataset (built into sklearn) to predict median house value from income, age, and location features. Always split before fitting and scale numeric inputs—regularized models are sensitive to feature scale.

from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_absolute_error, r2_score

X, y = fetch_california_housing(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

model = Pipeline([
    ("scale", StandardScaler()),
    ("reg", Ridge(alpha=1.0)),
])
model.fit(X_train, y_train)
pred = model.predict(X_test)

print("R²:", round(r2_score(y_test, pred), 3))
print("MAE:", round(mean_absolute_error(y_test, pred), 3))

Interpreting Coefficients

After scaling, each coefficient tells you the expected change in y per one standard-deviation increase in that feature, holding others fixed. Sign and magnitude matter; statistical significance is a separate question (use statsmodels or bootstrap if you need confidence intervals).

Engineering Habit — Baseline First

Before trying random forests or XGBoost, report linear regression metrics on the same split and features. If Ridge is within a few points of the complex model, prefer the simpler, faster, more interpretable solution.

When Linear Regression Struggles

Good Fit

Monotonic relationships (more sq ft → higher price)
Additive effects without sharp thresholds
Enough samples relative to feature count
Outliers handled or robust loss considered

Poor Fit

Strong nonlinear interactions (e.g., age × renovation)
Step functions and category-specific jumps
Heavy-tailed targets without log transform
Classification problems (use logistic regression instead)

Critical Mistake — Evaluating on Training Data

Perfect or near-perfect training R² often means overfitting or data leakage. Report test-set metrics from a split the model never saw during fit. Use cross-validation when data is limited.

Knowledge Check

Short Answer: What loss does ordinary least squares minimize? Answer: Sum of squared residuals between predictions and true values.
True/False: Ridge regression can eliminate features entirely by setting coefficients to exactly zero. Answer: False—that is Lasso; Ridge shrinks but rarely zeros coefficients.
Multiple Choice: Best first metric for regression on dollar amounts: (a) accuracy, (b) MAE, (c) silhouette score. Answer: (b).
Short Answer: Why scale features before Ridge? Answer: Penalty treats all coefficients equally; unscaled features with large ranges get unfairly penalized.
Short Answer: When should you try a nonlinear model instead? Answer: When residual plots show systematic curvature or test R² stalls well below business requirements.
True/False: R² of 1.0 on training always means a good production model. Answer: False—it may be overfit.
Multiple Choice: Ordinary least squares uses: (a) gradient boosting, (b) the normal equations / squared-error minimization, (c) k-NN. Answer: (b).
Short Answer: What does a residual plot help diagnose? Answer: Nonlinearity, heteroscedasticity, or outliers after the linear fit.
True/False: An intercept term is usually included in linear regression. Answer: True.
Multiple Choice: Collinear features make OLS: (a) more stable, (b) unstable with large coefficient variance, (c) sparse. Answer: (b).

Key Takeaways

Linear regression predicts continuous targets with interpretable weights.
Ridge and Lasso add regularization when you have many or correlated features.
Always evaluate on held-out data with MAE, RMSE, and R².
Scale numeric inputs for regularized linear models.
Next: Logistic Regression for binary and multiclass classification.

Trainer’s Guide

Hands-on idea: Students fit OLS, Ridge, and Lasso on California housing, plot predicted vs. actual on the test set, and write one sentence interpreting the top three coefficients.

Discussion prompt: Would you log-transform MedHouseVal before modeling? What changes in interpretation?

Recap: Linear regression predicts continuous targets with interpretable weights and least-squares loss. Continue with Logistic Regression.