Model Evaluation — Module 5.1

How This Lesson Fits the Module—and Volume 05

You defined datasets, features, and labels; split data; tuned with cross-validation; and packaged work in pipelines. Model evaluation asks: is this model good enough to ship—and good for the right reasons?

This lesson is the capstone of Module 5.1: ML Fundamentals. Master honest metrics here, then enter Module 5.2: Supervised Learning to study specific algorithms in depth.

Learning Objectives

By the end of this lesson, students should be able to:

Select classification and regression metrics aligned to business costs.
Compute and interpret precision, recall, F1, ROC-AUC, and MAE/RMSE/R².
Use classification_report and confusion_matrix in sklearn.
Explain why accuracy misleads on imbalanced labels.
Evaluate a fitted pipeline once on the locked test set.
Complete the Module 5.1 checklist before algorithm deep dives.

Metrics Match the Problem

A metric is a proxy for value. Fraud detection cares about recall on fraud; spam filters balance precision vs user annoyance; regression forecasts may prioritize MAE over MSE if large errors are not exponentially worse.

Task	Common metrics	When to emphasize
Binary classification	Precision, recall, F1, ROC-AUC, PR-AUC	Imbalanced classes; asymmetric error cost
Multiclass	Macro/micro F1, log loss	Rare classes need macro averaging
Regression	MAE, MSE, RMSE, R²	MAE robust to outliers; MSE penalizes large errors
Ranking	NDCG, MAP	Search and recommendations

Classification Evaluation

from sklearn.metrics import (
    classification_report, confusion_matrix, roc_auc_score
)

y_pred = pipe.predict(X_test)
y_proba = pipe.predict_proba(X_test)[:, 1]

print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))
print("ROC-AUC:", roc_auc_score(y_test, y_proba))

Critical Mistake — Accuracy on Imbalanced Data

99% accuracy sounds great when 98% of rows are negative—a dummy classifier that always predicts negative gets 98%. Always inspect per-class precision/recall and consider PR-AUC when positives are rare.

Regression Evaluation

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

y_pred = pipe.predict(X_test)
print("MAE:", mean_absolute_error(y_test, y_pred))
print("RMSE:", mean_squared_error(y_test, y_pred, squared=False))
print("R2:", r2_score(y_test, y_pred))

Beyond a Single Number

Slice evaluation

Metrics by region, product, or cohort
Catch fairness and drift issues early

Calibration

Do 70% predicted probabilities occur ~70% of the time?
Critical for thresholding and risk scoring

Cost-sensitive thresholds

Default 0.5 is rarely optimal
Tune threshold on validation, not test

Engineering Habit — Evaluation Report

Ship a one-page eval report: data snapshot version, split policy, CV best params, test metrics, confusion matrix, and pipeline artifact hash. Stakeholders need context—not a lone AUC number.

Capstone: Module 5.1 End-to-End Checklist

Module 5.1 built the discipline of trustworthy supervised learning workflows. Before Module 5.2 algorithms, confirm:

Data Contract

Dataset card with grain and label definition
Explicit X / y column lists
Volume 04 leakage review passed
Train/serve feature parity documented

Modeling Discipline

Train / validation / test roles respected
Split strategy matches time and groups
Full Pipeline in CV and search
Test evaluated once; artifact serialized

Students entering Logistic Regression and other Module 5.2 lectures should treat this checklist as non-negotiable. Algorithms change; the workflow does not.

Bridge to Module 5.2: Supervised Learning

Module 5.2 explores how specific estimators learn decision boundaries and ensembles. You already know how to split data, tune safely, pipeline transforms, and read metrics. The next step is understanding which algorithm fits your tabular problem—and why.

Knowledge Check

Short Answer: When is accuracy misleading? Answer: Strong class imbalance or unequal error costs.
True/False: Higher ROC-AUC always means better business outcomes. Answer: False—threshold and costs matter.
Multiple Choice: Penalizes large errors more: (a) MAE, (b) MSE, (c) MAPE always. Answer: (b).
Short Answer: What matrix shows TP/FP/FN/TN? Answer: Confusion matrix.
Short Answer: When should you use the test set? Answer: Once, after all tuning, for final unbiased estimate.
True/False: ROC-AUC requires a 0.5 classification threshold. Answer: False—it measures ranking across thresholds.
Multiple Choice: Retention team prioritizes catching churners: (a) precision, (b) recall, (c) row count. Answer: (b).
Short Answer: What does a false positive mean in churn? Answer: Predicted churn but customer stayed.
Short Answer: Name one Module 5.1 capstone checklist item. Answer: e.g., leakage review passed or pipeline serialized.
Multiple Choice: Capstone workflow last step before Module 5.2: (a) delete test labels, (b) eval report + serialized pipeline, (c) tune on test. Answer: (b).

Key Takeaways

Pick metrics that reflect business costs, not convenience.
Use precision/recall/F1 and ROC-AUC for classification; MAE/RMSE/R² for regression.
Evaluate slices and calibration before deployment.
Protect the test set; CV handles tuning.
Module 5.1 complete—continue to Module 5.2: Supervised Learning for algorithm deep dives.

Trainer’s Guide — Capstone Exercise

Hands-on idea: End-to-end churn capstone: dataset card, pipeline with CV tuning, single test evaluation, eval report with confusion matrix. Peer review checks for test peeking and missing pipeline serialization.

Discussion prompt: Stakeholders want “highest accuracy.” How do you negotiate a metric that matches fraud cost asymmetry?

Recap: Model evaluation picks metrics that match business cost and reports them on locked test data. Continue to Module 5.2 Supervised Learning.