Chunking — Module 14.1

How This Lesson Fits the Module & Volume

With an embedding model chosen, RAG still cannot embed a 200-page PDF as one vector and expect precise answers. Chunking decides the retrieval unit: small enough to be specific, large enough to be coherent. It sits between model choice and the mechanical splitting algorithms that implement those boundaries.

Chunk size also drives Vol. 13 token budgets: each retrieved chunk is input you pay for.

Learning Objectives

By the end of this lesson, students should be able to:

Define chunking as partitioning source documents into retrievable units.
Balance specificity, context, and embedding max length when sizing chunks.
Explain overlap and why boundary cuts can orphan answers.
Relate chunk design to top-k packing and cost-per-request.
Choose structure-aware chunks (sections, headings) over naive windows when possible.
Preview how splitting strategies operationalize chunking policy.

Definition

Chunking is the process of dividing source documents into smaller text units (chunks) that will be embedded, indexed, retrieved, and inserted into the LLM context as evidence.

The Chunk Size Tradeoff

Smaller chunks	Larger chunks
More precise retrieval hits	More surrounding context per hit
Risk of incomplete answers	Risk of diluted / noisy vectors
More vectors to store	Fewer vectors; thicker prompts
Fit short embedding windows	May truncate in the encoder

Fixed windows

Simple & predictable
Blind to structure
Needs overlap

Structure-aware

Respect headings / code
Better semantics
Needs parsers

Parent–child

Retrieve small child
Expand parent for LLM
Extra metadata

Code: Simple Overlapping Chunks

def chunk_text(text: str, size: int = 500, overlap: int = 100) -> list[str]:
    """Character windows — replace with token-aware sizing in production."""
    if overlap >= size:
        raise ValueError("overlap must be < size")
    chunks, start = [], 0
    while start < len(text):
        end = min(len(text), start + size)
        chunks.append(text[start:end])
        if end == len(text):
            break
        start = end - overlap
    return chunks

doc = "A" * 1200
parts = chunk_text(doc, size=400, overlap=80)
print(len(parts), [len(p) for p in parts])
# Attach metadata: source_id, chunk_ix, start/end offsets for citations.

Strengths of deliberate chunking

Improves retrieval precision
Keeps encoder inputs valid
Makes citations local & auditable

Tradeoffs

Bad boundaries lose answers
Overlap increases index size
One size rarely fits all doc types

Common Misconception

“Chunk size = context window size.” No. Chunks are retrieval units; the LLM context holds several chunks plus instructions. Size chunks for embedding quality and answer locality, then pack top-k under a separate token budget.

Knowledge Check

Short Answer: What is a chunk in RAG? Answer: A retrievable text unit embedded and stored in the index.
True/False: Larger chunks always improve RAG quality. Answer: False—they can dilute embeddings and waste tokens.
Multiple Choice: Overlap mainly helps with: (a) GPU cooling, (b) cutting mid-idea / boundary loss, (c) CSS. Answer: (b).
Short Answer: Why does chunking affect cost? Answer: Retrieved chunks become paid input tokens.
True/False: Structure-aware chunking ignores headings. Answer: False.
Multiple Choice: Parent–child retrieval retrieves: (a) small units then expands context, (b) only images, (c) CSS files. Answer: (a).
Short Answer: Name metadata to store with chunks. Answer: source_id, offsets, chunk index, title/section (any).
True/False: Chunk size should equal the full LLM context window. Answer: False.
Multiple Choice: Next lecture details: (a) Splitting, (b) Vol. 1 only, (c) printers. Answer: (a).
Short Answer: Risk of tiny chunks? Answer: Incomplete context / orphaned answers.

Key Takeaways

Chunking defines the unit of retrieval and citation.
Trade specificity against context; use overlap and structure wisely.
Size for the embedding model; pack for the LLM budget separately.
Next: Splitting.

Trainer’s Guide

Lab: Same handbook, three chunk sizes; measure recall@5 on 25 questions.

Prompt: When is parent–child worth the extra pipeline complexity?

Recap: Chunking is RAG’s information architecture. Continue with Splitting.