Generators & Iterators — Module 3.4

Essential Python Skills for AI Engineers (added — needed in practice, not in original outline)

How This Lesson Fits the Module

Training on millions of examples cannot load the entire dataset into RAM. Iterators provide a uniform way to traverse sequences one item at a time; generators are functions that produce items lazily using yield. PyTorch DataLoader, Python file reading, and streaming API responses all build on these ideas.

After *args and **kwargs, generators are the next building block for scalable AI pipelines—especially before you reach Volume 04’s data engineering topics.

Learning Objectives

By the end of this lesson, students should be able to:

Distinguish iterables, iterators, and generator objects.
Write generator functions with yield and generator expressions.
Explain why generators use constant memory for large sequences.
Use next(), iter(), and for loops with iterators correctly.
Recognize generators in data loading, log streaming, and batch construction.
Avoid exhausting a generator by iterating it twice without re-creating it.

Introduction: Lazy Sequences

A list materializes every element in memory. A generator produces values on demand—one at a time—pausing after each yield and resuming when the consumer asks for the next value.

Definition

Iterable — object you can loop over (list, str, file).
Iterator — object with __next__() that returns the next item or raises StopIteration.
Generator — iterator created by a function containing yield or by a generator expression.

def read_batches(paths, batch_size=32):
    batch = []
    for path in paths:
        with open(path) as f:
            for line in f:
                batch.append(line.strip())
                if len(batch) == batch_size:
                    yield batch
                    batch = []
    if batch:
        yield batch

This generator never holds more than one batch in memory, even if paths references terabytes of text.

Generator Expressions

Like list comprehensions but lazy: (expression for item in iterable) returns a generator, not a list.

lengths = (len(t) for t in token_stream)  # no list allocated
total = sum(lengths)  # consumes the generator once

AI Example — Streaming Records

Processing a 10 GB JSONL file line by line:

import json

def iter_jsonl(path):
    with open(path) as f:
        for line in f:
            yield json.loads(line)

for record in iter_jsonl("train.jsonl"):
    if record.get("label") == 1:
        process(record)

Approach	Memory	When to Use
List	O(n) for all items	Small collections, repeated passes
Generator	O(1) extra beyond current item	Large files, infinite streams
NumPy / Pandas	Columnar, chunked options	Numeric tables, EDA

Common Misconception: “You can loop over a generator multiple times.”

Reality: Generators are single-pass. After exhaustion, they raise StopIteration. Call the generator function again or materialize to a list if you need multiple iterations.

Knowledge Check

True/False: yield pauses a function and saves its state. Answer: True.
Short Answer: (x*x for x in range(5)) returns what type? Answer: generator object.
Multiple Choice: Best for reading a 50 GB log file: (a) readlines(), (b) line iterator / generator, (c) list comprehension, (d) global variable. Answer: (b).
Short Answer: What exception signals iterator exhaustion? Answer: StopIteration.
Short Answer: Difference between an iterable and an iterator? Answer: An iterable can be looped over; an iterator has __next__() and yields the next item.
True/False: You can iterate a generator twice without recreating it. Answer: False—generators are single-pass; call the function again or materialize a list.
Multiple Choice: Memory cost of a generator over a huge stream is typically: (a) O(n) for all items, (b) O(1) extra beyond the current item, (c) O(n²), (d) unbounded RAM always. Answer: (b).
Short Answer: What keyword creates a generator function? Answer: yield.
True/False: A batching generator can hold only one batch in memory while reading many files. Answer: True.
Multiple Choice: Best when you need repeated random access over a small collection: (a) generator only, (b) list, (c) infinite stream, (d) StopIteration. Answer: (b).

Key Takeaways

Generators produce items lazily with yield.
They enable memory-efficient pipelines over large data.
Generator expressions are the lazy cousin of comprehensions.
Next: Decorators, which often wrap generators and API calls.

Trainer’s Guide

Memory demo: Compare sys.getsizeof(list(range(10**6))) vs a generator. The list allocates; the generator does not.

Recap: Generators yield values lazily so large datasets stay memory-efficient; next, wrap functions with Decorators.