Pipeline — Module 5.1

How This Lesson Fits the Module

Volume 04 and every split lesson warned: fit transforms on training data only. A sklearn Pipeline enforces that mechanically by chaining preprocessing and model into one estimators you fit, predict, and serialize once.

For AI engineers, the pipeline is the deployable unit—the same object in notebooks, batch retrains, and joblib artifacts behind your API.

Learning Objectives

By the end of this lesson, students should be able to:

Build Pipeline and ColumnTransformer chains for tabular data.
Explain why pipelines prevent train/test leakage from preprocessing.
Use pipelines with cross_val_score and GridSearchCV.
Serialize and load models with joblib for serving.
Inspect intermediate steps with named_steps when debugging.
Treat the pipeline as the single contract between train and production.

Pipeline Anatomy

A Pipeline lists (name, transformer_or_estimator) tuples. All steps except the last must implement fit and transform; the final step is a predictor with fit and predict.

Component	Role in pipeline
`SimpleImputer`	Fill missing values (fit on train)
`StandardScaler`	Scale numerics (fit on train)
`OneHotEncoder`	Encode categoricals (fit on train)
`ColumnTransformer`	Apply different prep per column group
Classifier / regressor	Final estimator

End-to-End Tabular Pipeline

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.ensemble import RandomForestClassifier

num_cols = ["tenure_days", "orders_last_90d", "avg_order_value"]
cat_cols = ["plan_tier", "country"]

preprocess = ColumnTransformer(
    transformers=[
        ("num", Pipeline([
            ("imputer", SimpleImputer(strategy="median")),
            ("scaler", StandardScaler()),
        ]), num_cols),
        ("cat", Pipeline([
            ("imputer", SimpleImputer(strategy="most_frequent")),
            ("ohe", OneHotEncoder(handle_unknown="ignore")),
        ]), cat_cols),
    ]
)

model = Pipeline([
    ("prep", preprocess),
    ("clf", RandomForestClassifier(n_estimators=200, random_state=42)),
])

model.fit(X_train, y_train)
y_pred = model.predict(X_test)

Critical Mistake — Separate Pickles for Prep and Model

Saving scaler and model in two files risks version skew at deploy. Serialize one pipeline object; serving calls pipeline.predict(raw_features) only.

Pipelines in CV and Hyperparameter Search

Pipelines compose with every tool in this module. Hyperparameters target steps via double underscores: clf__max_depth, prep__num__imputer__strategy (when needed).

import joblib
from sklearn.model_selection import cross_val_score

scores = cross_val_score(model, X_train, y_train, cv=5, scoring="roc_auc")

joblib.dump(model, "artifacts/churn_model_v3.joblib")
loaded = joblib.load("artifacts/churn_model_v3.joblib")
loaded.predict(X_test[:5])

Without Pipeline

Manual fit/transform ordering
Easy to leak test stats
Train/serve skew risk

With Pipeline

One fit / predict API
CV-safe by construction
Single serialized artifact

Engineering Habit — Pipeline Schema Test

CI test: fit pipeline on sample rows, dump, load, assert predictions match on frozen fixture. Catches sklearn version bumps and column order regressions.

Knowledge Check

Short Answer: Why use a Pipeline? Answer: Chains prep and model; prevents leakage and simplifies deploy.
True/False: Only the final pipeline step may have predict. Answer: True for standard predict API.
Multiple Choice: Leakage-safe CV requires: (a) prep outside CV, (b) full pipeline inside CV, (c) test fit on imputer. Answer: (b).
Short Answer: What does ColumnTransformer do? Answer: Runs different transformers on different column subsets.
Short Answer: Why one joblib file? Answer: Guarantees prep and model versions stay matched in production.
True/False: model.named_steps["clf"] accesses the classifier. Answer: True.
Multiple Choice: handle_unknown="ignore" belongs in: (a) RandomForest, (b) OneHotEncoder, (c) StandardScaler. Answer: (b).
Short Answer: Volume 04 connection to pipelines? Answer: Pipelines enforce fit-on-train-only for all learned transforms.
Short Answer: Nested Pipeline inside ColumnTransformer — why? Answer: Multiple steps per column group (impute then encode).
Multiple Choice: Production inference should call: (a) scaler.transform then model.predict manually, (b) pipeline.predict on raw features, (c) refit on request. Answer: (b) with a fitted pipeline.

Key Takeaways

Pipelines bundle preprocessing and model into one leakage-safe estimator.
ColumnTransformer handles mixed tabular dtypes cleanly.
Serialize the whole pipeline for train/serve parity.
Next: Model Evaluation—module capstone tying splits, metrics, and ship criteria together.

Trainer’s Guide

Hands-on idea: Students break a notebook that fits scaler globally, then rebuild with Pipeline + joblib round-trip test asserting identical predictions.

Discussion prompt: Your API receives JSON features. Where should JSON→DataFrame conversion live relative to the pickled pipeline?

Recap: A pipeline bundles preprocessing and the estimator so transforms fit only on training data. Continue with Model Evaluation.