Splitting — Module 14.1

How This Lesson Fits the Module & Volume

Chunking is the policy; splitting is the algorithm that applies it—by characters, tokens, sentences, Markdown headers, or recursive separators. Good splitters preserve meaning at boundaries and keep chunks under the embedding model’s max length.

Once splits exist, retrieval can search them.

Learning Objectives

By the end of this lesson, students should be able to:

Distinguish chunking policy from splitting algorithms.
Implement recursive separator splits (paragraph → sentence → word).
Prefer token-aware lengths over raw characters when counting for models.
Handle code and Markdown with structure-preserving separators.
Record offsets so citations map back to source documents.
Avoid silent mid-token / mid-table cuts that break meaning.

Definition

Splitting is the concrete procedure that cuts a document string into chunk strings according to size limits and separator preferences (e.g., recursive character/token splitting, sentence splitting, or heading-based splits).

Common Split Strategies

Strategy	Cuts on	Best for
Fixed window	N chars/tokens + overlap	Quick baselines
Recursive separators	\n\n, \n, “. ”, space	General prose
Sentence	NLP sentence boundaries	QA-friendly prose
Markdown / HTML	Headings, sections	Docs sites, handbooks
Code-aware	Functions / classes	Repos / API refs

Character count

Easy to code
Mismatch with tokens
OK for demos

Token count

Matches model limits
Needs a tokenizer
Production default

Semantic split

Embedding breakpoints
Costlier to build
Advanced corpora

Code: Recursive Separator Split

def split_recursive(text: str, max_len: int = 800, separators=None) -> list[str]:
    separators = separators or ["\n\n", "\n", ". ", " ", ""]
    if len(text) <= max_len:
        return [text] if text.strip() else []
    sep = next((s for s in separators if s in text), "")
    if sep == "":
        return [text[i:i + max_len] for i in range(0, len(text), max_len)]
    parts, chunks = text.split(sep), []
    buf = ""
    for part in parts:
        candidate = part if not buf else buf + sep + part
        if len(candidate) <= max_len:
            buf = candidate
        else:
            if buf:
                chunks.extend(split_recursive(buf, max_len, separators[1:]))
            buf = part
    if buf:
        chunks.extend(split_recursive(buf, max_len, separators[1:]))
    return chunks

sample = "Intro para.\n\nDetails about refunds.\nMore detail. Extra sentence."
print(split_recursive(sample, max_len=40))

Strengths

Respects natural breakpoints
Configurable per document type
Composable with overlap post-pass

Tradeoffs

Recursive logic can surprise
Tables/lists still hard
Must validate max length hard-caps

Common Misconception

“Any splitter is fine if average length looks right.” Average length hides pathological chunks: one 2k-token blob and dozens of 20-token scraps. Inspect length histograms and spot-check boundaries on real PDFs and Markdown.

Knowledge Check

Short Answer: How does splitting differ from chunking? Answer: Chunking is the policy/unit design; splitting is the algorithm that cuts text.
True/False: Recursive splitting tries larger separators first. Answer: True (typically \n\n before spaces).
Multiple Choice: Token-aware splitting is better because: (a) matches model limits, (b) looks prettier, (c) removes GPUs. Answer: (a).
Short Answer: Why store character offsets? Answer: Map citations back to the source document.
True/False: Code should use the same separators as novels. Answer: False—use code-aware boundaries.
Multiple Choice: Heading-based splits suit: (a) structured docs, (b) only audio, (c) CSS-only sites. Answer: (a).
Short Answer: What should you plot after splitting? Answer: Chunk length histogram / distribution.
True/False: Mid-table splits are harmless. Answer: False—they break meaning.
Multiple Choice: Next topic: (a) Retrieval, (b) Vol. 1 only, (c) printers. Answer: (a).
Short Answer: Name one separator used in recursive prose splits. Answer: Blank line, newline, period+space, or space.

Key Takeaways

Splitters implement chunking with separators and size caps.
Prefer token-aware, structure-preserving cuts.
Validate length distributions and citation offsets.
Next: Retrieval.

Trainer’s Guide

Lab: Split one Markdown handbook three ways; students vote which boundaries look answerable.

Prompt: How would you split a CSV vs a legal contract differently?

Recap: Splitting turns policy into reproducible cuts. Continue with Retrieval.