Pandas — Module 3.3

How This Lesson Fits the Module

NumPy gives you fast arrays. Real AI projects rarely start with clean tensors—they start with CSV exports, JSON logs, database dumps, and spreadsheets. Pandas is the workhorse for loading, cleaning, joining, and exploring tabular data before it becomes model input.

Every sklearn pipeline and every feature-engineering notebook begins with a DataFrame. Master Pandas here; you will use it daily as an AI engineer.

Learning Objectives

By the end of this lesson, students should be able to:

Load CSV and Parquet files into DataFrame objects.
Select, filter, group, and merge datasets for ML feature construction.
Handle missing values, duplicates, and type coercion safely.
Convert DataFrames to NumPy arrays for model training.
Decide when Pandas is the right tool versus SQL, Polars, or Spark.
Profile data quality issues before training models.

What Pandas Is—and When to Use It

Pandas adds labeled, heterogeneous tables (DataFrame) and aligned series (Series) on top of NumPy. Column names, row indices, and datetime indexes make exploratory data analysis (EDA) and feature engineering practical at laptop scale.

Use Pandas when…	Use something else when…
EDA on datasets that fit in RAM (< few GB)	Data exceeds single-machine memory → Spark, Dask, Polars lazy
Joining multiple tables for feature stores	Heavy transactional queries → SQL in the warehouse
Time-series resampling and rolling windows	Real-time streaming → Kafka + stream processors
Quick CSV/Excel/JSON ingestion in notebooks	Production serving → typed pipelines + feature stores

Loading and Inspecting Data

The first step in any ML project: understand what you actually have. Never train on data you have not profiled.

import pandas as pd

df = pd.read_csv("customer_churn.csv")
print(df.shape)
print(df.dtypes)
print(df.isna().sum())        # missing values per column
print(df.describe())          # numeric summary stats
df.head()

Engineering Habit — The Data Contract

Before modeling, document expected columns, dtypes, null rates, and value ranges. A five-line assert block or Great Expectations check prevents silent schema drift from breaking production pipelines.

Selection, Filtering, and Feature Engineering

Pandas supports intuitive label-based (loc) and integer-based (iloc) indexing. Boolean masks express business rules as code.

# Feature engineering for a churn model
df["tenure_months"] = df["tenure_days"] / 30.44
df["is_high_value"] = (df["monthly_spend"] > df["monthly_spend"].median()).astype(int)

features = ["tenure_months", "monthly_spend", "support_tickets", "is_high_value"]
X = df.loc[df["tenure_months"] >= 1, features]   # drop brand-new accounts
y = df.loc[X.index, "churned"]

GroupBy Aggregations

Aggregate signals—mean session length per user, total purchases per category—are classic ML features. groupby + agg replaces hundreds of lines of SQL-like logic.

user_features = (
    events.groupby("user_id")
    .agg(
        event_count=("event_id", "count"),
        avg_session_sec=("session_sec", "mean"),
        last_seen=("timestamp", "max"),
    )
    .reset_index()
)

Merging Tables

Production datasets are fragmented. Join user demographics to event logs to product catalogs—the same relational patterns databases use, now in Python.

training = users.merge(user_features, on="user_id", how="left")
training = training.merge(product_catalog, on="product_id", how="inner")
training["avg_session_sec"] = training["avg_session_sec"].fillna(0)

Missing Values and Leakage

Critical Mistake — Target Leakage

Filling missing values using global statistics computed on the entire dataset—including the test set—leaks future information. Fit imputers on training data only, then transform validation and test sets. Scikit-learn’s Pipeline enforces this pattern.

# Simple imputation sketch (fit on train only in production)
median_spend = train_df["monthly_spend"].median()
train_df["monthly_spend"] = train_df["monthly_spend"].fillna(median_spend)

# Bridge to NumPy / sklearn
import numpy as np
X_train = train_df[features].to_numpy(dtype=np.float32)
y_train = train_df["churned"].to_numpy()

Pandas Strengths

Fast iteration on tabular EDA
Rich I/O for CSV, Parquet, JSON, SQL
Time-series and categorical tooling
Seamless handoff to NumPy and sklearn

Pandas Limitations

Single-node memory ceiling
Performance cliffs on very wide or very long data
Not a production feature-serving layer
Chained assignment can surprise beginners

Knowledge Check

Short Answer: Difference between loc and iloc? Answer: loc uses labels; iloc uses integer positions.
True/False: groupby followed by agg can compute multiple statistics per group. Answer: True.
Short Answer: Why call .to_numpy() before sklearn? Answer: sklearn expects numeric arrays, not DataFrames with mixed dtypes.
Multiple Choice: Safest join when you need every user even without events: (a) inner, (b) left, (c) right. Answer: (b).
Short Answer: What is target leakage? Answer: Using information from the test/future set during training preprocessing.
Short Answer: Which methods profile a newly loaded DataFrame? Answer: e.g. shape, dtypes, isna().sum(), describe(), head().
True/False: Pandas DataFrames add labeled columns and row indexes on top of NumPy. Answer: True.
Multiple Choice: Data that exceeds single-machine RAM is better handled with: (a) only Pandas, (b) Spark/Dask/Polars lazy, (c) Matplotlib, (d) a text file. Answer: (b).
Short Answer: What does groupby + agg typically produce for ML? Answer: Aggregated features such as counts, means, or last-seen timestamps per entity.
True/False: Fit imputers on the full dataset including the test set. Answer: False—fit on training data only, then transform val/test.

Key Takeaways

Pandas is the standard tool for tabular data loading, cleaning, and feature engineering.
Profile dtypes, nulls, and distributions before modeling.
Fit transformations on training data only to avoid leakage.
Export clean numeric matrices with .to_numpy() for downstream ML libraries.
Next: Matplotlib to visualize what your data actually looks like.

Trainer’s Guide

Hands-on idea: Give students a messy CSV with mixed date formats and nulls. Grade them on a reproducible cleaning notebook that outputs a sklearn-ready matrix.

Discussion prompt: When would you switch from Pandas to Polars or Spark? What are the memory and latency trade-offs?

Recap: Pandas loads, cleans, and engineers tabular features; next, visualize those tables with Matplotlib.