The training set teaches the model; the testing set is the honest exam on unseen rows. It answers: “If we deployed today on new customers, how well would we do?”
AI engineers treat the test set like a sealed envelope—touch it once at the end, or metrics become optimistically biased from repeated peeking.
Learning Objectives
By the end of this lesson, students should be able to:
- Define the testing set and its role in unbiased performance estimation.
- Contrast testing with training and validation purposes.
- Evaluate held-out data with
score,predict, and metrics. - Explain why repeated test-set tuning invalidates results.
- Guard test rows from preprocessing leakage (Volume 04).
- Report test results with business context, not only accuracy.
Purpose of the Test Set
The testing set is labeled data never used during training or hyperparameter search. It simulates production: the model sees X_test without y_test influencing any earlier decision.
| Split | Used for | How often to use |
|---|---|---|
| Training | Learn parameters | Every fit |
| Validation | Compare models / tune knobs | Many iterations |
| Test | Final generalization estimate | Once (per major release) |
Evaluating on X_test
After final model selection, refit on train+validation (optional) or train only per team policy, then predict test. Preprocessing must be fit on training data only, then applied to test.
Tuning features because test ROC dropped, then re-evaluating on the same test set, overfits the test split. Use validation or cross-validation for iteration; reserve test for sign-off.
When Test Metrics Mislead
Random splits on time-series, grouped users, or shifting markets produce test sets that do not match deployment. Align split strategy with how the model will be used live.
| Scenario | Test pitfall | Better approach |
|---|---|---|
| User-level churn | Same user in train and test | Group split by user_id |
| Demand forecasting | Random shuffle | Time-based test window |
| Rare fraud | Test with zero positives | Stratified split |
Store test row IDs in a read-only bucket. CI fails if training scripts load them. Document the single approved evaluation notebook allowed to read y_test.
Knowledge Check
- Short Answer: Why hold out a test set? Answer: Unbiased estimate of performance on unseen data.
- True/False: It is fine to tune hyperparameters using test accuracy. Answer: False—use validation or CV.
- Multiple Choice: Test set should be used: (a) every epoch, (b) once for final evaluation, (c) for imputation means. Answer: (b).
- Short Answer: What is test-set contamination? Answer: Test information influenced training or tuning decisions.
- Short Answer: Difference between validation and test? Answer: Validation supports iteration; test is final locked evaluation.
- True/False: Scaling fit on train+test is acceptable for test evaluation. Answer: False—leakage.
- Multiple Choice: Best split for monthly sales forecast: (a) random 80/20, (b) last 3 months as test, (c) duplicate rows. Answer: (b).
- Short Answer: What does
predict_probaprovide? Answer: Class probability scores for classification. - Short Answer: Why stratify test split on imbalanced
y? Answer: Preserve class proportions in test for meaningful metrics. - Multiple Choice: After changing model because test failed, you should: (a) reuse same test, (b) collect new held-out data, (c) delete labels. Answer: (b) ideally; acknowledge reused test is optimistic.
Key Takeaways
- The test set is the final unbiased check before shipping.
- Never tune on test metrics; guard against leakage in preprocessing.
- Split design must mirror production (time, groups, balance).
- Next: Validation Set—where iteration happens safely.
Hands-on idea: Run two experiments: one tunes on validation, one peeks at test. Compare how test metrics diverge from validation when students “cheat.”
Discussion prompt: Stakeholders want weekly test re-runs on the same split to track progress. What do you push back on?
Recap: The test set is a locked holdout for a final unbiased estimate of generalization. Continue with Validation Set.