Data Leakage — Module 4.1

How This Lesson Fits the Module—and Volume 04

You have collected, cleaned, labeled, engineered, augmented, imputed, and screened data. Data leakage is the silent failure that can make all of it meaningless: models look brilliant offline and fail in production because future or forbidden information slipped into training.

This lesson is the capstone of Volume 04: Data Engineering for AI. Master leakage prevention here, then enter Volume 05: Machine Learning with pipelines you can trust.

Learning Objectives

By the end of this lesson, students should be able to:

Define data leakage and explain why inflated offline metrics are a liability.
Identify train/test split leakage from preprocessing and duplicate records.
Prevent temporal leakage in forecasting and event-driven datasets.
Recognize target leakage from proxy features and post-outcome columns.
Structure sklearn Pipeline + cross_val_score to keep transforms honest.
Apply a pre-deployment checklist before training models in Volume 05.

What Data Leakage Is

Data leakage occurs when information that would not be available at prediction time influences model training or evaluation. The model is not learning general patterns—it is memorizing shortcuts. Leakage is worse than overfitting: you often will not see the problem until live traffic arrives.

Leakage type	What goes wrong	Example
Train/test leakage	Test statistics enter training	Scaling on full dataset before split
Temporal leakage	Future informs the past	Random split on time-series transactions
Target leakage	Features encode the label	`refund_issued` predicting `churn`
Duplicate leakage	Same entity in train and test	Two photos of one user in different splits

Critical Mistake — Chasing 99% Accuracy

When validation accuracy jumps from 72% to 99% after a new feature, suspect leakage before celebrating. Real-world lifts are usually incremental. Ask: “Could I know this feature at scoring time?”

Train/Test Leakage

Any transformation that learns from data—imputation means, scaler variance, vocabulary, target encoding, feature selection—must be fit on the training split only. Nested inside each cross-validation fold, the same rule applies.

# WRONG — scaler sees test distribution
scaler = StandardScaler().fit(X)          # all data
X_train, X_test = train_test_split(X)

# RIGHT — pipeline fit only on training rows
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

preprocess = StandardScaler()

pipe = Pipeline([
    ("prep", preprocess),
    ("clf", LogisticRegression()),
])

scores = cross_val_score(pipe, X, y, cv=5, scoring="roc_auc")
print(scores.mean())   # each fold fits prep on train folds only

Temporal Leakage

When rows are ordered in time—user events, sales, sensor readings—a random shuffle split places future transactions in training while the model is evaluated on the past. Metrics look strong; deployment walks backward through time.

from sklearn.model_selection import TimeSeriesSplit

tscv = TimeSeriesSplit(n_splits=5)
for train_idx, test_idx in tscv.split(X):
    X_tr, X_te = X.iloc[train_idx], X.iloc[test_idx]
    y_tr, y_te = y.iloc[train_idx], y.iloc[test_idx]
    # Features for row t may only use events <= t (point-in-time joins)

Engineering Habit — Point-in-Time Features

In warehouses, build features with AS OF joins: for each label timestamp, aggregate only prior events. Replay pipelines on historical dates to verify no future columns appear.

Target Leakage

Target leakage sneaks in features that are consequences of the label or collected only after the outcome is known. They are deadly because they are highly predictive and perfectly useless live.

Task	Leaky feature (avoid)	Legitimate alternative
Predict loan default	`collections_calls_after_default`	Prior delinquency counts before application
Predict churn	`account_closed_date`	Declining usage trend in prior 30 days
Diagnose disease from intake	`treatment_prescribed`	Symptoms at admission only
Forecast demand	Same-day shipped quantity	Lagged sales and promotions

Target encoding, label filtering, and global deduplication are frequent hidden sources. If a feature’s definition references the label window, stop and redesign.

Duplicate and Group Leakage

Splitting rows without grouping by user_id, patient_id, or document_id puts near-identical records in both train and test. Image datasets with burst photos of one object suffer the same issue. Use GroupKFold or group-aware splits.

Volume 04 Checklist

Schema + feature catalog reviewed
Missingness and outlier rules in pipelines
All transforms fit on train / CV folds only
Time-aware splits for sequential data
No post-outcome columns in feature matrix

Bridge to Volume 05

Start with sklearn Pipeline baselines
Pick metrics aligned to business cost
Hold out a final test set untouched until the end
Serialize prep + model as one artifact
Monitor live drift against training distributions

Capstone: Trustworthy Data-to-Model Handoff

Volume 04 built the discipline of reliable inputs. Volume 05 teaches algorithms—but algorithms cannot fix leaked signal. Your handoff document should answer:

What is the prediction moment (what time, what event)?
Which columns are guaranteed available then?
How were train, validation, and test splits constructed?
Where does preprocessing live (single serialized pipeline)?

Students entering Module 5.1: ML Fundamentals should treat this checklist as non-negotiable. The first models you train—logistic regression, decision trees, cross-validation—assume the data contract you define here.

Knowledge Check

Short Answer: Define data leakage. Answer: Training or evaluation uses information unavailable at real prediction time.
True/False: Random train/test split is always valid for time-series. Answer: False—use temporal splits.
Multiple Choice: Leakiest feature for churn prediction: (a) tenure_days, (b) support_tickets_last_30d, (c) cancellation_confirmation_email_sent. Answer: (c).
Short Answer: Why use Pipeline inside cross_val_score? Answer: Preprocessing refits per fold on training folds only.
Short Answer: What is group leakage? Answer: Related records (same user) appear in both train and test.
True/False: Scaling the full dataset before splitting is a common leakage bug. Answer: True.
Multiple Choice: Target leakage is: (a) using future or post-outcome columns as features, (b) too few trees, (c) missing values. Answer: (a).
Short Answer: What is a point-in-time feature? Answer: A value knowable at the prediction timestamp, not after the outcome.
True/False: Duplicate rows across train and test can inflate accuracy. Answer: True.
Multiple Choice: Honest CV requires: (a) global preprocessing then split, (b) Pipeline + per-fold fit, (c) test labels in training. Answer: (b).

Key Takeaways

Leakage produces optimistic offline metrics and production failures.
Fit every learned transform on training data; use pipelines and proper CV.
Time-series and event data need temporal splits and point-in-time features.
Reject features that encode post-outcome or label-proxy information.
Volume 04 complete—continue to Volume 05: Machine Learning to train models on data you trust.

Trainer’s Guide — Capstone Exercise

Hands-on idea: Give students a leaky notebook (global scaler + random time split + suspicious feature). They debug metrics, fix the pipeline, and write a one-page data contract for Volume 05 modeling.

Discussion prompt: Your stakeholder wants to include “customer satisfaction survey after purchase.” Walk through whether and how it can ever be a legitimate feature.

Recap: Leakage inflates offline metrics by using information unavailable at prediction time. Continue to Vol. 05 Dataset.