Data Collection

Introduction: The First Mile of ML

Every model is trained on observations someone chose to record. Collection defines the population your model will generalize to—and the biases it will inherit. Before you call pd.read_csv, answer: who created this data, for what purpose, and is it legal to use for your task?

Production AI teams separate raw (immutable landing zone) from curated (cleaned, labeled, feature-ready). This lecture focuses on getting data into raw storage reliably and traceably.

Data Source Landscape

Collecting from Files and APIs

Bridge Volume 03 skills into a repeatable ingest pattern: fetch or copy, validate shape, write to a dated raw path, log metadata.

from datetime import datetime, timezone
from pathlib import Path
import hashlib
import json
import os

import pandas as pd
import requests

RAW = Path("data/raw/support_tickets")
RAW.mkdir(parents=True, exist_ok=True)

def ingest_csv_export(src: Path, source_id: str) -> dict:
    df = pd.read_csv(src, encoding="utf-8")
    ts = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
    out = RAW / f"{source_id}_{ts}.parquet"
    df.to_parquet(out, index=False)
    digest = hashlib.sha256(out.read_bytes()).hexdigest()
    meta = {
        "source_id": source_id,
        "collected_at": ts,
        "rows": len(df),
        "columns": list(df.columns),
        "path": str(out),
        "sha256": digest,
    }
    with open(RAW / f"{source_id}_{ts}.meta.json", "w") as f:
        json.dump(meta, f, indent=2)
    return meta

def fetch_paginated_api(base_url: str, headers: dict, page_param: str = "page"):
    page = 1
    rows = []
    while True:
        resp = requests.get(
            base_url,
            headers=headers,
            params={page_param: page, "per_page": 100},
            timeout=30,
        )
        resp.raise_for_status()
        batch = resp.json()["data"]
        if not batch:
            break
        rows.extend(batch)
        page += 1
    return pd.DataFrame(rows)

Public Datasets and Hubs

Benchmark datasets accelerate prototyping, but production models often need proprietary or domain-specific data. When using public hubs, verify license, documentation, and demographic coverage.

# Hugging Face — stream large splits without loading all RAM
from datasets import load_dataset

ds = load_dataset("imdb", split="train", streaming=True)
sample = next(iter(ds))
print(sample.keys(), sample["text"][:80])

Web Scraping: Capability and Responsibility

Use robots.txt as a signal, not legal advice. Prefer official APIs and licensed datasets for production ML. Regex from Module 3.4 helps extract fields from semi-structured HTML only when parsing is permitted.

Collection Checklist for AI Engineers

• Task alignment — Does each row map to a prediction or generation objective?
• Temporal split — Can you hold out future data to simulate deployment?
• PII scan — Regex or dedicated tools before sharing notebooks.
• Idempotent jobs — Re-runs should not corrupt prior raw snapshots.
• Cost model — API tokens, warehouse scan bytes, storage growth.

Knowledge Check