Chunk-Level Token Planning — Module 13.3

How This Lesson Fits the Module & Volume

RAG and long-doc pipelines live or die by chunk size in tokens. Chunks too large waste the window; too small shatter meaning. Planning happens before indexing, not after a failed call.

Reconnects Vol. 11 context window limits with Vol. 12 tiktoken counting and later Vol. 14 RAG.

Learning Objectives

By the end of this lesson, students should be able to:

Choose chunk token targets from window and top-k packing math.
Leave headroom for system, query, and output reserve.
Overlap chunks intentionally without blowing budgets.
Build an indexer that cuts on tiktoken boundaries.
Estimate tokens_per_request = query + k * chunk + overhead.
Revisit chunk size when the model window or k changes.

Definition

Chunk-level token planning is sizing and overlapping document segments so that retrieval packs a predictable token footprint inside the model context window.

Parameter	Role	Typical starting point
chunk_tokens	Index unit size	256–512 for many RAG apps
overlap_tokens	Boundary continuity	10–20% of chunk
top_k	Chunks per query	3–8
pack_budget	Max tokens for docs	Window − system − query − out

Code: Plan Pack Budget

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")

WINDOW, SYS, QUERY, OUT = 128_000, 2_000, 500, 2_000
pack_budget = WINDOW - SYS - QUERY - OUT
top_k = 6
chunk_tokens = pack_budget // top_k
print("pack_budget", pack_budget, "chunk_tokens<=", chunk_tokens)

def chunk_by_tokens(text: str, size: int, overlap: int):
    ids = enc.encode(text)
    out, i = [], 0
    while i < len(ids):
        out.append(enc.decode(ids[i:i+size]))
        i += max(size - overlap, 1)
    return out

Fixed token chunks

Stable packing math
May split sentences
Simple ops

Structure-aware

Headings/sections
Variable sizes
Better coherence

Hybrid

Soft max tokens + structure
More code
Production default

Strengths

Predictable request shapes
Easier cost forecasts
Fewer surprise overflows

Tradeoffs

Bad sizes hurt recall
Overlap multiplies storage
Must retune per model

Common Misconception

“500 characters ≈ 500 tokens, so our old splitter is fine.” Character heuristics drift by language and code. Plan and split in token units for the embedding/chat models you use.

Knowledge Check

Short Answer: What is pack_budget? Answer: Tokens left for retrieved docs after system/query/output reserves.
True/False: top_k and chunk_tokens jointly determine doc tokens in a call. Answer: True.
Multiple Choice: Overlap mainly helps: (a) GPU clock, (b) boundary continuity, (c) CSS layout. Answer: (b).
Short Answer: Why replan when window changes? Answer: Pack math and optimal chunk size shift.
True/False: Character chunking equals token chunking. Answer: False.
Multiple Choice: Best counter for OpenAI chat packing: (a) tiktoken, (b) abacus only, (c) DPI. Answer: (a).
Short Answer: Name a risk of huge chunks. Answer: Wasted window / diluted retrieval / overflow.
Short Answer: Name a risk of tiny chunks. Answer: Lost context / fragmented meaning.
Multiple Choice: Vol. 11 idea behind the ceiling: (a) context window, (b) dropout, (c) batch norm. Answer: (a).
True/False: Chunk planning is only an embedding concern, never chat. Answer: False—chat packing depends on it.

Key Takeaways

Size chunks from pack_budget and top_k.
Split on token boundaries.
Overlap deliberately; measure storage cost.
Retune when models or k change.
Next: Tokenizer Tools.

Trainer’s Guide

Lab: For a 32k model and k=5, compute max chunk_tokens; implement chunk_by_tokens on a sample PDF text.

Discussion: Should embedding model and chat model share chunk sizes? When not?

Recap: Chunk planning makes RAG token-predictable. Continue with Tokenizer Tools.