Dataset — Module 5.1

How This Lesson Fits the Module—and Volume 04

In Data Leakage you learned to protect train, validation, and test splits from forbidden information. Volume 05 begins where that handoff ends: turning a dataset—a structured collection of examples—into something an algorithm can learn from.

A dataset is not just a CSV on disk. For AI engineers it is a data contract: row grain, column definitions, label window, and the prediction moment. Every split, pipeline, and metric in Module 5.1 assumes you can describe your dataset precisely.

Learning Objectives

By the end of this lesson, students should be able to:

Define a machine learning dataset as rows (samples) and columns (variables).
Distinguish raw warehouse tables from model-ready training tables.
Load and inspect tabular data with pandas using dtypes, shape, and head.
Document dataset grain, time range, and class balance for stakeholders.
Identify leakage risks inherited from Volume 04 before any modeling.
Prepare a minimal X / y frame for downstream lessons.

What a Dataset Is in ML Engineering

A dataset is a collection of observations where each row is one example (customer, transaction, image, document) and each column is a measured attribute. Models do not train on “the database”—they train on a curated matrix extracted for a specific task with a defined label.

Concept	Engineering meaning	Example
Sample / instance	One row the model learns from	Single subscription as of 2025-06-01
Feature column	Input available at prediction time	`tenure_days`, `plan_tier`
Label column	Outcome to predict (often withheld at scoring)	`churned_30d`
Metadata	IDs and timestamps for joins—usually not features	`user_id`, `snapshot_date`

Engineering Habit — Dataset Card

Ship a one-page dataset card with every modeling project: source tables, SQL grain, date range, row count, label definition, known biases, and Volume 04 leakage checklist status. Future you (and auditors) will need it.

Loading and Profiling with pandas

Before algorithms, engineers profile data: shape, dtypes, missingness, duplicates, and label distribution. sklearn expects numeric matrices or pandas DataFrames with consistent column order between train and serve.

import pandas as pd

df = pd.read_parquet("s3://ml-artifacts/churn/churn_training_2025Q2.parquet")

print(df.shape)                    # (rows, columns)
print(df.dtypes)
print(df.isna().mean().sort_values(ascending=False).head())
print(df["churned_30d"].value_counts(normalize=True))

# Separate metadata from modeling columns
meta_cols = ["user_id", "snapshot_date"]
feature_cols = [c for c in df.columns if c not in meta_cols + ["churned_30d"]]
X = df[feature_cols]
y = df["churned_30d"]

Raw Tables vs Model-Ready Datasets

Volume 04 ETL and feature engineering produce wide tables. A model-ready dataset narrows that table: one row per prediction unit, point-in-time features only, label aligned to the business question, and metadata stripped before fit.

Stage	Typical location	Who consumes it
Raw ingest	Lake / warehouse landing zone	Data engineers
Curated features	Feature store or mart	Analytics + ML
Training export	Versioned parquet/CSV snapshot	Training jobs
Serving features	Online store / batch scorer	Production API

Critical Mistake — Training on the Wrong Grain

If each user appears in ten rows but churn is a user-level label, you do not have ten independent samples—you have repeated measures. Fix grain before split lessons or metrics will lie.

Bridging Volume 04: Trust Before Training

Carry forward the leakage capstone checklist: prediction moment defined, no post-outcome columns, transforms fit on train only, temporal splits when needed. Your dataset export should reference that checklist in its README or dataset card.

Volume 04 Deliverable

Clean, labeled, engineered columns
Leakage review complete
Feature catalog with definitions

Module 5.1 Starts Here

Define X and y explicitly
Profile class balance and missingness
Version the snapshot used for training

Knowledge Check

Short Answer: What is one row in a tabular ML dataset? Answer: One sample or instance (e.g., one customer at one snapshot).
True/False: Any warehouse table is ready to pass directly to fit. Answer: False—grain, labels, and leakage must be resolved first.
Multiple Choice: Best first step after loading data: (a) train XGBoost, (b) profile shape/dtypes/missingness, (c) delete half the rows. Answer: (b).
Short Answer: Why version training snapshots? Answer: Reproducibility and auditability when models are retrained.
Short Answer: What connects this lesson to Data Leakage? Answer: The dataset must exclude forbidden future or post-outcome information.
Multiple Choice: user_id is usually: (a) a feature, (b) metadata for grouping, (c) the label. Answer: (b).
True/False: Class imbalance in y affects metric choice later. Answer: True.
Short Answer: What does df.shape return? Answer: Tuple of (number of rows, number of columns).
Short Answer: Define model-ready dataset. Answer: Curated table with correct grain, valid features, and aligned labels for one task.
Multiple Choice: Label column in churn example: (a) tenure_days, (b) churned_30d, (c) snapshot_date. Answer: (b).

Key Takeaways

A dataset is a task-specific matrix of samples and columns—not the entire warehouse.
Profile with pandas before modeling; document grain and label definition.
Volume 04 leakage discipline applies to every export you train on.
Next: Feature—which columns become model inputs.

Trainer’s Guide

Hands-on idea: Students receive a messy multi-table dump and must produce a dataset card plus a single parquet with explicit X / y columns and a written grain statement.

Discussion prompt: Your dataset has 2% positive labels. How does that influence business metrics and modeling choices in later lectures?

Recap: A dataset is the labeled (or unlabeled) table your model learns from; its quality bounds every later metric. Continue with Feature.