Missing Values — Module 4.1

How This Lesson Fits the Module

Real datasets are incomplete. After Data Augmentation expands training diversity, you still need principled handling of missing values—nulls, blank fields, sensor dropouts, and implicit gaps that break models and skew metrics.

Imputation is not neutral: the strategy you choose encodes assumptions about why data is missing. Done wrong, it becomes a subtle form of data leakage.

Learning Objectives

By the end of this lesson, students should be able to:

Classify missingness mechanisms: MCAR, MAR, and MNAR.
Audit missing rates, patterns, and correlations with the target.
Choose among deletion, simple imputation, model-based, and multivariate strategies.
Add missingness indicator features when absence is informative.
Fit imputers inside sklearn pipelines on training data only.
Document imputation choices for reproducibility and compliance.

Why Missing Values Break ML Pipelines

Most algorithms require complete numeric input. Even tree libraries that accept NaN behave differently than explicit imputation strategies. Pandas NaN, empty strings, 99999 sentinels, and NULL in SQL are all missing-data problems disguised as different types.

Mechanism	Meaning	Example
MCAR (Missing Completely At Random)	Missingness unrelated to any variable	Random packet loss in logs
MAR (Missing At Random)	Missingness depends on observed columns	Income missing more often for younger users (age observed)
MNAR (Missing Not At Random)	Missingness depends on the hidden value itself	High earners skip income field

Engineering Habit — Missingness Report

Before imputing, publish a missingness report: percent null per column, co-occurrence heatmap, and correlation between is_missing flags and the target. If missingness predicts the label, treat absence as a feature.

Exploratory Analysis

import pandas as pd
import numpy as np

# Treat empty strings as missing for object columns
df.replace("", np.nan, inplace=True)

missing_pct = df.isna().mean().sort_values(ascending=False)
print(missing_pct.head(10))

# Missingness indicator (often high signal)
for col in ["income", "credit_score"]:
    df[f"{col}_was_missing"] = df[col].isna().astype(int)

Imputation Strategies

Strategy	When to use	Risk
Listwise deletion (`dropna`)	Very low missing rate; MCAR	Biased samples; wastes rows
Constant / sentinel	Categorical “Unknown” bucket	Sentinel collides with real values
Mean / median / mode	Quick baseline for low missing %	Shrinks variance; ignores correlations
Group-wise imputation	MAR with known segments	Sparse groups fall back poorly
k-NN imputation	Correlated numeric features	Slow on wide data; needs scaling
MICE / iterative	Multivariate numeric tables	Complex; still assumes MAR
Model-based (RF, regression)	Strong predictors of missing col	Must nest inside CV to avoid leakage

from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier

num_imputer = SimpleImputer(strategy="median")
cat_imputer = SimpleImputer(strategy="most_frequent")

preprocess = ColumnTransformer([
    ("num", num_imputer, num_cols),
    ("cat", cat_imputer, cat_cols),
])

clf = Pipeline([
    ("impute_and_encode", preprocess),
    ("model", RandomForestClassifier()),
])

# Fit on train only
clf.fit(X_train, y_train)
# clf.predict(X_test) uses train-derived statistics

Critical Mistake — Global Imputation Before Split

Computing df["age"].fillna(df["age"].median()) on the full dataframe leaks test-set distribution into training. Split first, fit imputer on X_train, then transform test. sklearn Pipeline + cross_val_score enforce this automatically.

Categorical and Text Missing Values

For categoricals, prefer an explicit "Unknown" level over mode imputation when missingness exceeds ~5%. For text, empty documents may mean “no comment”—impute to empty string or a token like [NO_TEXT] consistently in train and serve paths.

When Deletion Is OK

<1–2% missing and MCAR plausible
Row is unusable without key identifier
Duplicate or corrupt record
Exploratory slice, not production training

When Impute + Indicator

Missingness correlates with target
MNAR suspected (add flag feature)
Regulatory need to retain all rows
Feature is expensive to collect later

Production Considerations

Serialize imputer statistics (joblib) with the model. At serve time, apply the same medians, modes, and category maps learned offline. Version imputation logic with the model artifact; silent schema changes are a top cause of production drift.

Knowledge Check

Short Answer: MCAR vs MNAR? Answer: MCAR is unrelated to any values; MNAR depends on the missing value itself.
True/False: Median imputation on the full dataset before train/test split is safe. Answer: False—it leaks test distribution.
Multiple Choice: Income missing more for high earners is: (a) MCAR, (b) MAR, (c) MNAR. Answer: (c).
Short Answer: Why add income_was_missing? Answer: Missingness itself may predict the target.
Short Answer: What does SimpleImputer(strategy="most_frequent") do for categoricals? Answer: Fills nulls with the most common category seen during fit.
True/False: Listwise deletion is always safe when missingness is MNAR. Answer: False—it can bias the remaining sample.
Multiple Choice: MAR means missingness depends on: (a) the missing value itself only, (b) observed variables, (c) nothing. Answer: (b).
Short Answer: Why put SimpleImputer inside a sklearn Pipeline? Answer: It fits on training folds only and reuses the same statistics at inference.
True/False: k-NN imputation can leak if neighbors include test rows. Answer: True.
Multiple Choice: Best first diagnostic for missingness: (a) drop all nulls silently, (b) profile null rates by column and segment, (c) fill with zero everywhere. Answer: (b).

Key Takeaways

Classify missingness (MCAR/MAR/MNAR) before choosing a strategy.
Simple mean/median imputation is a baseline, not a default for every column.
Missingness indicators turn absence into a legitimate feature.
Fit imputers on training data only; bundle them in pipelines for production parity.
Next: Outlier Detection to separate signal from noise and fraud.

Trainer’s Guide

Hands-on idea: Provide a dataset with 15% structured missingness. Students compare listwise deletion, median imputation, and k-NN imputation inside cross-validation and justify their production choice.

Discussion prompt: A healthcare field is often blank when clinicians rush. Is that MCAR, MAR, or MNAR? How does that change your approach?

Recap: Classify missingness, impute inside train-only pipelines, and consider missingness indicators. Continue with Outlier Detection.