Feature Engineering — Module 4.1

How This Lesson Fits the Module

ETL moves raw records into a usable table. Feature engineering is where you turn those columns into signals a model can learn—ratios, encodings, time windows, and domain-specific composites that often matter more than algorithm choice.

Strong features built on clean data are the bridge between data preparation and model training in Volume 05: Machine Learning. Invest here before chasing fancier architectures.

Learning Objectives

By the end of this lesson, students should be able to:

Explain why feature engineering often outperforms model tuning on tabular problems.
Encode categorical variables with one-hot, ordinal, and target encoding (with leakage awareness).
Derive datetime, ratio, and interaction features from raw columns.
Scale and normalize numeric features appropriately for different algorithms.
Compose sklearn Pipeline and ColumnTransformer steps that fit on training data only.
Document feature definitions so production serving matches offline training.

What Feature Engineering Is—and Why It Matters

Feature engineering is the deliberate construction of input variables from raw data. Models do not see “business reality”—they see numbers in a matrix. Your job is to make those numbers informative, stable, and available at prediction time.

Raw column	Engineered feature	Why it helps
`order_total`, `item_count`	`avg_item_price = order_total / item_count`	Captures basket composition beyond totals alone
`signup_date`	`tenure_days`, `is_weekend_signup`	Seasonality and lifecycle effects
`city` (high cardinality)	Frequency encoding or region bucket	Reduces sparsity while preserving signal
`text_review`	Length, sentiment score, keyword flags	Compresses unstructured text into tabular signals

Engineering Habit — Feature Catalog

Maintain a living feature catalog: name, SQL/Python definition, dtype, null handling, and whether it is available in real time. When offline accuracy jumps but production metrics flatline, the catalog is where you discover train/serve skew.

Encoding Categorical Variables

Tree models can sometimes handle label-encoded integers; linear models and neural nets on tabular data usually need careful encoding. Choose based on cardinality, ordinality, and leakage risk.

Method	Best for	Caution
One-hot encoding	Low cardinality (< ~20 levels)	Wide sparse matrices; rare levels in test set
Ordinal encoding	True order (e.g., shirt size S–XL)	Never assign arbitrary integers to nominal categories
Target / mean encoding	High cardinality with enough samples	Must use out-of-fold or nested CV—direct encoding leaks the target
Frequency encoding	Popularity signals (city, SKU)	Rare categories need an `unknown` bucket

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

df["avg_order_value"] = df["lifetime_spend"] / df["order_count"].clip(lower=1)
df["days_since_last_order"] = (pd.Timestamp.today() - df["last_order_date"]).dt.days

num_cols = ["tenure_days", "avg_order_value", "days_since_last_order"]
cat_cols = ["plan_tier", "country"]

preprocess = ColumnTransformer(
    transformers=[
        ("num", StandardScaler(), num_cols),
        ("cat", OneHotEncoder(handle_unknown="ignore"), cat_cols),
    ]
)

clf = Pipeline([
    ("prep", preprocess),
    ("model", LogisticRegression(max_iter=1000)),
])

Datetime and Rolling Features

Time is not a single column—it is a generator of features. Extract components, compute recency, and aggregate behavior over windows. Always compute windows relative to a cutoff that respects causality (no future events in training features).

events["event_hour"] = events["timestamp"].dt.hour
events["is_weekend"] = events["timestamp"].dt.dayofweek >= 5

# Point-in-time feature: orders in prior 30 days per user
cutoff = pd.Timestamp("2025-06-01")
window_start = cutoff - pd.Timedelta(days=30)
user_30d = (
    orders.loc[(orders["order_date"] >= window_start) & (orders["order_date"] < cutoff)]
    .groupby("user_id")["order_id"]
    .count()
    .rename("orders_last_30d")
)

Scaling and Transformations

Distance-based models (k-NN, SVM, neural nets) need scaled inputs. Tree ensembles are scale-invariant but still benefit when features span orders of magnitude. Log transforms tame heavy-tailed spend or latency distributions.

Common Scalers

StandardScaler — zero mean, unit variance
MinMaxScaler — bounded [0, 1] range
RobustScaler — uses median/IQR; resists outliers
FunctionTransformer — log1p, sqrt, custom

When Trees Need Less

Random Forest / XGBoost splits on thresholds
Monotonic transforms preserve split order
Still encode categoricals and fix units
Interactions may matter more than scaling

Critical Mistake — Fitting on the Full Dataset

Computing StandardScaler statistics, target encodings, or vocabulary maps on train+test data leaks information. Fit every transformer inside a cross-validation fold or on the training split only, then transform validation and test sets.

Feature Selection and Dimensionality

More features are not always better. Remove constants, near-duplicates, and columns with >95% missing values unless missingness itself is informative. Use domain knowledge first; use SelectKBest, L1 regularization, or permutation importance as secondary filters.

Knowledge Check

Short Answer: Why create avg_order_value instead of feeding raw spend and count? Answer: It captures per-order behavior and may be more stable than either column alone.
True/False: Target encoding computed on the full dataset before splitting is safe. Answer: False—it leaks target information.
Multiple Choice: Best scaler when outliers dominate spend: (a) MinMaxScaler, (b) RobustScaler, (c) no scaling. Answer: (b).
Short Answer: What does handle_unknown="ignore" on OneHotEncoder do? Answer: Maps unseen categories to all-zero vectors instead of raising an error.
Short Answer: Why document features in a catalog? Answer: Ensures training and serving use identical definitions and catches skew early.
True/False: Random Forest always needs StandardScaler. Answer: False—tree splits are threshold-based; scaling is optional.
Multiple Choice: ColumnTransformer is used to: (a) train neural nets, (b) apply different transforms to numeric vs categorical columns, (c) split data. Answer: (b).
Short Answer: Why fit transformers only on training data? Answer: Test statistics would leak into the model and inflate offline metrics.
True/False: One-hot encoding high-cardinality IDs is usually a good default. Answer: False—it explodes dimensionality; consider target encoding or hashing.
Multiple Choice: Useful datetime features often include: (a) only unix timestamp, (b) hour/day-of-week/month (plus cyclic encodings), (c) random noise. Answer: (b).

Key Takeaways

Feature engineering translates domain knowledge into model-readable signals.
Encode categoricals deliberately; high-cardinality fields need leakage-safe strategies.
Datetime and rolling aggregates are among the highest-ROI tabular features.
Fit all transformers on training data only—use sklearn pipelines to enforce this.
Next: Data Augmentation to expand limited datasets for vision and NLP.

Trainer’s Guide

Hands-on idea: Provide a retail CSV with timestamps and categories. Students deliver a sklearn Pipeline with at least five engineered features and a written feature catalog entry for each.

Discussion prompt: Which features would be impossible to compute in real-time serving? How would you redesign or precompute them?

Recap: Feature engineering turns domain knowledge into model-ready columns via encodings, ratios, and leakage-safe pipelines. Continue with Data Augmentation.