Regular Expressions (re) — Module 3.4

Essential Python Skills for AI Engineers (added — needed in practice, not in original outline)

How This Lesson Fits the Module — Capstone

This is the final lecture in Module 3.4 and closes Volume 03’s Python arc. You have learned comprehensions, decorators, file I/O, APIs, and async patterns. Regular expressions (regex) are the bridge to Volume 04: Data Engineering for AI—where raw text becomes structured, validated, and pipeline-ready data.

Regex powers log parsing, PII redaction, data validation, entity extraction heuristics, and preprocessing before tokenization. Every data engineer and AI engineer encounters patterns that plain string methods cannot express cleanly.

Learning Objectives

By the end of this lesson, students should be able to:

Explain what regular expressions match: patterns, not literal strings only.
Use Python’s re module: search, match, findall, sub.
Apply common metacharacters: ., \d, \w, +, *, ?, [], ().
Extract capture groups and named groups from matched text.
Choose between regex and simpler tools (split, replace, parsers).
Apply regex to data-engineering tasks: validation, cleaning, and field extraction.

Introduction: Patterns in Text

A regular expression is a compact language for describing string patterns. Instead of checking if "@" in email with ad-hoc logic, you define a pattern that describes valid structure—then search, extract, or replace matches across millions of rows.

In AI pipelines, regex appears in ingestion scripts (extract dates from filenames), guardrails (detect credit-card-like numbers), and ETL transforms (normalize phone formats) before data reaches Pandas or a vector database.

Definition — Python re Module Basics

import re

text = "Contact: [email protected] or [email protected]"

# findall — all matches
re.findall(r"[\w.+-]+@[\w.-]+\.\w+", text)

# search — first match anywhere
m = re.search(r"support@([\w.]+)", text)
m.group(1)  # capture group: acme.ai

# sub — replace
re.sub(r"\d{3}-\d{2}-\d{4}", "[SSN REDACTED]", log_line)

Essential Metacharacters

Pattern	Meaning	Example
`\d`	Digit	`\d+` matches `42`
`\w`	Word character (letter, digit, _)	`\w+` matches `user_01`
`\s`	Whitespace	`\s+` splits fields
`.`	Any character (except newline)	`a.c` matches `abc`
`+` / `*` / `?`	One or more / zero or more / optional	`colou?r` matches color/colour
`[abc]`	Character class	`[A-Z]{2}` state codes
`(...)`	Capture group	Extract substrings

Data Engineering Example — Parse Log Lines

import re

PATTERN = re.compile(
    r"^(?P<ts>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) "
    r"\[(?P<level>\w+)\] "
    r"(?P<msg>.*)$"
)

line = "2026-07-13 14:22:01 [ERROR] Embedding API timeout"
m = PATTERN.match(line)
if m:
    record = m.groupdict()
    # {"ts": "...", "level": "ERROR", "msg": "..."}

Named groups turn unstructured logs into dict records ready for JSONL export—a pattern you will extend in Volume 04 data preparation pipelines.

Regex vs Other Tools

Use Regex

Pattern-based validation (email-ish, IDs)
Extract fields from semi-structured text
Redact or normalize repeating formats

Use Alternatives

HTML/XML → proper parsers (BeautifulSoup, lxml)
JSON → json.loads
Complex grammars → parser generators (pyparsing, Lark)
Semantic extraction → LLMs with structured output

Bridge to Volume 04 Regex is one step in the data preparation chain: ingest raw text → extract fields → validate → transform → load. Volume 04 covers the full pipeline—schema design, Parquet, orchestration, and quality checks—building on the Python foundations you completed in Volume 03.

Common Misconception: “Regex is perfect for validating email addresses and HTML.”

Reality: Full RFC-compliant email regex is notoriously unwieldy. Use regex for pragmatic checks; use dedicated validators and parsers when correctness is critical.

Common Misconception: “Greedy matching is always what you want.”

Reality: Quantifiers are greedy by default (.* consumes as much as possible). Use non-greedy .*? or more specific patterns to avoid over-matching.

Flags and Readability

# re.IGNORECASE — case-insensitive
re.findall(r"error", text, flags=re.IGNORECASE)

# re.compile — reuse pattern for performance in loops
EMAIL = re.compile(r"[\w.+-]+@[\w.-]+\.\w+")
EMAIL.findall(document)

Quick Knowledge Check

Short Answer: What does \d{4} match? Answer: Exactly four digits.
Write: Regex to find words starting with “data”. Answer: r"\bdata\w*" or similar.
True/False: re.findall returns all non-overlapping matches. Answer: True.
Multiple Choice: Parse valid JSON from an API: (a) regex, (b) json.loads, (c) both equally, (d) neither. Answer: (b).
Short Answer: Why compile patterns with re.compile in a loop over millions of lines? Answer: Avoids re-parsing the pattern each iteration.
Short Answer: What does re.sub do? Answer: Replaces matches of a pattern with a substitution string.
True/False: .* is greedy by default. Answer: True.
Multiple Choice: Extract named fields from log lines: (a) split() only, (b) named capture groups, (c) random.choice, (d) set union. Answer: (b).
Short Answer: Difference between re.match and re.search? Answer: match anchors at start; search finds first match anywhere.
Multiple Choice: Volume 04 focuses on: (a) regex only, (b) data engineering pipelines, (c) GPU kernels, (d) HTML design. Answer: (b).

Key Takeaways

Regex describes patterns for search, extraction, and replacement.
Python’s re module provides search, findall, sub, and compile.
Use capture groups to structure semi-structured text for pipelines.
Prefer dedicated parsers when formats have real grammars (JSON, HTML).
Module 3.4 complete. Continue to Volume 04: Data Engineering for AI.

Trainer’s Guide

Capstone exercise: Given a folder of messy support tickets, extract email addresses, ticket IDs (TKT-\d+), and dates into a CSV. Preview the Volume 04 mindset: raw → structured → export.

Discussion: When would you replace regex extraction with an LLM? Trade-offs: cost, determinism, latency, maintainability.

What’s Next Open Volume 04: Data Engineering for AI and begin Module 4.1 Data Preparation.