Outlier Detection — Module 4.1

How This Lesson Fits the Module

After handling Missing Values, the next quality gate is outlier detection—identifying points that deviate so far from the bulk distribution that they distort models, metrics, or business decisions.

Not every outlier is an error. Some are fraud, VIP customers, or rare events you must predict. The skill is detection first, then a domain decision: cap, remove, model separately, or keep.

Learning Objectives

By the end of this lesson, students should be able to:

Distinguish univariate, multivariate, and contextual outliers.
Apply the IQR rule and z-score method with correct assumptions.
Explain when robust statistics outperform mean-based thresholds.
Introduce Isolation Forest for multivariate anomaly detection.
Choose between clipping (winsorization), removal, and robust modeling.
Detect outliers on training data only to avoid peeking at test distribution.

What Makes a Point an Outlier?

An outlier is an observation that is inconsistent with the majority of the data under a chosen model of normality. Context matters: $50k monthly spend is normal for enterprise accounts and impossible for free-tier users.

Type	Definition	Example
Univariate	Extreme on one column	Height = 280 cm in adult cohort
Multivariate	Unusual combination of features	Low income + luxury car purchases
Contextual	Normal globally, abnormal in segment	Ice cream sales spike in winter for one store
Collective	Entire subgroup diverges	Sensor batch drift over a week

IQR Method (Robust Univariate)

The interquartile range method uses quartiles, so a few extreme values do not pull the threshold the way mean ± 3σ does. Classic rule: flag points below Q1 − 1.5×IQR or above Q3 + 1.5×IQR.

import numpy as np
import pandas as pd

def iqr_outlier_mask(series: pd.Series, k: float = 1.5) -> pd.Series:
    q1 = series.quantile(0.25)
    q3 = series.quantile(0.75)
    iqr = q3 - q1
    lower = q1 - k * iqr
    upper = q3 + k * iqr
    return (series < lower) | (series > upper)

df["spend_outlier"] = iqr_outlier_mask(df["monthly_spend"])

Engineering Habit — Plot Before Dropping

Box plots and log-scale histograms take five minutes and prevent deleting legitimate tail revenue. Document every removal rule; auditors and teammates will ask.

Z-Score Method

The z-score measures how many standard deviations a point lies from the mean. Values with |z| > 3 (or 2.5) are often flagged. This assumes approximate normality and is sensitive to the very outliers you are trying to find.

from scipy import stats

z = np.abs(stats.zscore(df["monthly_spend"], nan_policy="omit"))
df["spend_z_outlier"] = z > 3

# Modified z-score (MAD) — more robust
median = df["monthly_spend"].median()
mad = np.median(np.abs(df["monthly_spend"] - median))
modified_z = 0.6745 * (df["monthly_spend"] - median) / mad
df["spend_mad_outlier"] = np.abs(modified_z) > 3.5

Method	Strengths	Weaknesses
IQR	Robust, interpretable, no normality assumption	Univariate only; fixed k is arbitrary
Z-score	Familiar, fast	Mean/std distorted by outliers; needs near-Gaussian data
Modified z (MAD)	Robust univariate alternative	Still one dimension at a time
Isolation Forest	Multivariate, few hyperparameters	Less interpretable; needs tuning contamination

Isolation Forest—Multivariate Anomalies

Isolation Forest builds random trees that isolate points; anomalies are easier to separate and require fewer splits. It scales well and handles correlated features better than applying IQR column-by-column.

from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

outlier_pipe = Pipeline([
    ("scale", StandardScaler()),
    ("iso", IsolationForest(
        n_estimators=200,
        contamination=0.01,   # expected outlier fraction — tune with domain input
        random_state=42,
    )),
])

# Fit on training features only (unsupervised)
outlier_pipe.fit(X_train)

# -1 = outlier, 1 = inlier
train_flags = outlier_pipe.predict(X_train)
X_train_clean = X_train[train_flags == 1]

Critical Mistake — Deleting Test Outliers Using Train+Test

Never refit IQR bounds or Isolation Forest on the combined dataset. Thresholds learned with test data leak distribution information. Fit detection on training data; apply the same rules (or frozen model) to validation and test.

What to Do After Detection

Remediation Options

Winsorize / clip — cap at percentiles
Remove — data entry errors only
Transform — log1p for heavy tails
Separate model — fraud vs normal users

When to Keep Outliers

They are the positive class (fraud, defects)
Business cares about tail events
Robust models (trees, Huber loss) handle them
Sample size is already small

Outliers vs. Missing Values vs. Leakage

Outlier handling interacts with imputation: mean imputation pulls toward the center and can mask extremes. Aggressive removal before cross-validation can leak label information if outliers correlate with the target. Treat outlier policy as part of the preprocessing pipeline, fit inside each CV fold.

Knowledge Check

Short Answer: IQR outlier bounds formula? Answer: Below Q1 − 1.5×IQR or above Q3 + 1.5×IQR.
True/False: Z-scores are robust when data is heavy-tailed. Answer: False—extremes inflate std and hide themselves.
Multiple Choice: Best first pass for multivariate fraud: (a) IQR per column, (b) Isolation Forest, (c) drop top 1% by mean. Answer: (b).
Short Answer: What does contamination=0.01 assume? Answer: Roughly 1% of points are outliers.
Short Answer: Why winsorize instead of delete? Answer: Retains sample size while limiting influence of extremes.
True/False: Isolation Forest should be fit on the full train+test matrix. Answer: False—fit on training data only.
Multiple Choice: A contextual outlier is: (a) extreme in any setting, (b) unusual given time or segment, (c) always a data error. Answer: (b).
Short Answer: Why prefer median/IQR over mean/std for univariate thresholds? Answer: They are robust to the extremes you are trying to detect.
True/False: Fraud outliers should usually be deleted before modeling. Answer: False—they may be the positive class.
Multiple Choice: RobustScaler uses: (a) mean and std, (b) median and IQR, (c) min and max. Answer: (b).

Key Takeaways

Outliers are context-dependent; domain review beats blind deletion.
IQR is a robust univariate rule; z-scores need near-normal data.
Isolation Forest extends detection to multivariate tabular features.
Fit outlier rules on training data; clip, transform, or model tails deliberately.
Next: Data Leakage—the capstone risk that invalidates all prior work.

Trainer’s Guide

Hands-on idea: Inject 2% synthetic fraud rows into a credit dataset. Students compare IQR per feature vs Isolation Forest recall and discuss false positives.

Discussion prompt: Your CEO wants “clean data” by removing top 5% spenders. What business metric might that destroy?

Recap: Detect outliers with context-aware rules, fit thresholds on train only, and treat tails deliberately. Continue with Data Leakage.