File I/O (JSON, CSV, text) — Module 3.4

Essential Python Skills for AI Engineers (added — needed in practice, not in original outline)

How This Lesson Fits the Module

Every ML project reads and writes data: training sets on disk, experiment configs, model outputs, evaluation CSVs. Module 3.3 introduced Pandas for tabular analysis; this lecture covers the foundational file I/O layer—text files, JSON (APIs and configs), and CSV (spreadsheets and exports) using the Python standard library.

Before Volume 04’s data engineering pipelines, you need reliable patterns for loading datasets and persisting results.

Learning Objectives

By the end of this lesson, students should be able to:

Read and write text files with open() and context managers (with).
Serialize and deserialize JSON with json.load / json.dump.
Read and write CSV files with the csv module.
Choose appropriate encodings (encoding="utf-8") for international text.
Handle common file paths with pathlib.Path.
Combine file I/O with exception handling for production scripts.

Introduction: Data on Disk

Models rarely live in isolation. Training scripts load JSON configs, iterate JSONL corpora, export CSV metrics, and write plain-text logs. Python’s built-in modules handle these formats without extra dependencies.

Text Files

from pathlib import Path

path = Path("logs/run_001.txt")
path.parent.mkdir(parents=True, exist_ok=True)

with path.open("w", encoding="utf-8") as f:
    f.write("epoch=1, loss=0.42\n")

with path.open("r", encoding="utf-8") as f:
    for line in f:
        print(line.strip())

Always use with (context manager) so files close even if an error occurs. Always specify encoding="utf-8" for portability.

JSON

Definition — JSON in Python

JSON maps to Python types: objects → dict, arrays → list, strings → str, numbers → int/float, booleans → bool, null → None.

import json

config = {"model": "bert-base", "lr": 2e-5, "epochs": 3}

with open("config.json", "w") as f:
    json.dump(config, f, indent=2)

with open("config.json") as f:
    loaded = json.load(f)

# JSONL: one JSON object per line (common for LLM datasets)
with open("train.jsonl") as f:
    records = [json.loads(line) for line in f if line.strip()]

CSV

import csv

rows = [{"id": 1, "label": "spam"}, {"id": 2, "label": "ham"}]

with open("labels.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=["id", "label"])
    writer.writeheader()
    writer.writerows(rows)

with open("labels.csv", newline="") as f:
    reader = csv.DictReader(f)
    for row in reader:
        print(row["label"])

Format	Best For	Python Module
Plain text	Logs, prompts, raw corpora	`open()`
JSON / JSONL	Configs, API payloads, LLM datasets	`json`
CSV	Tabular exports, spreadsheet interchange	`csv` or Pandas

Common Misconception: “Use CSV for everything because Excel opens it.”

Reality: CSV struggles with nested structures, commas in text, and schema evolution. JSON/JSONL is standard for LLM fine-tuning data; Parquet (Volume 04) is better for large analytics tables.

Knowledge Check

Short Answer: Why use with open(...)? Answer: Ensures the file is closed automatically.
True/False: JSONL stores one JSON object per line. Answer: True.
Multiple Choice: Config file for a training run: (a) CSV, (b) JSON, (c) BMP image, (d) WAV audio. Answer: (b).
Short Answer: Why specify encoding="utf-8" when opening text files? Answer: Portability for international text across platforms.
True/False: JSON objects map to Python dicts and arrays to lists. Answer: True.
Short Answer: Which stdlib types write CSV with headers from dict rows? Answer: csv.DictWriter (and DictReader to read).
Multiple Choice: Large LLM fine-tuning corpora are typically stored as: (a) BMP, (b) JSONL, (c) WAV only, (d) Excel macros. Answer: (b).
True/False: CSV is ideal for nested API payloads and evolving schemas. Answer: False—JSON/JSONL handles nested structures better.
Short Answer: How does pathlib.Path help before writing logs? Answer: Build paths and create parent directories (e.g. mkdir(parents=True, exist_ok=True)).
Multiple Choice: For large analytics tables later in Volume 04, prefer: (a) CSV only, (b) Parquet, (c) screenshots, (d) markdown tables. Answer: (b).

Key Takeaways

Use context managers and UTF-8 encoding for text files.
JSON powers configs and API data; JSONL scales to large corpora.
CSV suits flat tabular exports; use Pandas for analysis.
Next: Virtual Environments & pip for reproducible project dependencies.

Trainer’s Guide

Pipeline mini-lab: Read JSONL records, filter by label, write a CSV summary. Connects I/O formats to a realistic preprocessing step.

Recap: Use context managers, UTF-8, JSON/JSONL, and CSV for configs and datasets; next, isolate dependencies with Virtual Environments & pip.