Truncation Strategies — Module 13.3

How This Lesson Fits the Module & Volume

When content exceeds the budget, truncation decides what survives. Bad truncation drops the instruction that mattered; good truncation is policy-driven and reversible in logs.

Uses tiktoken to cut on token boundaries—not mid-byte guesses—and respects Vol. 11 context window math.

Learning Objectives

By the end of this lesson, students should be able to:

Compare head, tail, middle, and priority-based truncation.
Truncate on token IDs, then decode safely.
Preserve system/safety text ahead of optional docs.
Log what was dropped for debugging.
Avoid naive character slices that break Unicode/BPE.
Combine truncation with summarization when loss is too high.

Definition

Truncation strategies are rules for deleting or compressing tokens so a payload fits a limit while minimizing task damage.

Strategy	Keeps	Best for
Head (prefix)	Start of text	Instructions first; rare for docs
Tail (suffix)	Most recent end	Logs, chat, newest facts
Middle omit	Head + tail	Long docs with thesis + conclusion
Priority / structured	Tagged must-keep fields	Multi-part prompts

Code: Token-Safe Tail Truncation

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")

def truncate_tail(text: str, max_tokens: int) -> str:
    ids = enc.encode(text)
    if len(ids) <= max_tokens:
        return text
    return enc.decode(ids[-max_tokens:])  # keep newest tokens

def truncate_middle(text: str, max_tokens: int, head_ratio=0.4) -> str:
    ids = enc.encode(text)
    if len(ids) <= max_tokens:
        return text
    head = int(max_tokens * head_ratio)
    tail = max_tokens - head
    return enc.decode(ids[:head] + ids[-tail:])

Truncate

Deterministic size
Information loss
Cheap CPU

Summarize then fit

More signal density
Extra model call / drift
Good for history

Reject

No silent loss
Needs UX
Safest for legal text

Strengths

Keeps calls succeeding under load
Policy can encode product priorities
Token-boundary safe with tiktoken

Tradeoffs

Silent loss if not logged
Middle cuts can break sentences
Wrong strategy wrecks tasks

Common Misconception

“Cutting to 8,000 characters is good enough.” Character cuts ignore {BPE} boundaries and multilingual density. Always truncate in token space for the target model.

Knowledge Check

Short Answer: Why truncate on token IDs? Answer: Matches model limits and avoids broken encoding slices.
True/False: Tail truncation keeps the newest tokens. Answer: True.
Multiple Choice: Safety system text should usually be: (a) first to drop, (b) last to drop, (c) randomly dropped. Answer: (b).
Short Answer: Name one alternative to truncation. Answer: Summarization, RAG, or reject/upload-split.
True/False: Character truncation is encoding-safe for all languages. Answer: False.
Multiple Choice: Middle-omit keeps: (a) only vowels, (b) head and tail, (c) only embeddings. Answer: (b).
Short Answer: Why log dropped spans? Answer: Debug wrong answers caused by missing context.
Short Answer: Which library from Vol. 12 supports encode/decode truncation? Answer: tiktoken.
Multiple Choice: For live chat history, prefer: (a) head-only ancient messages, (b) newest-first/tail policy, (c) shuffle. Answer: (b).
True/False: Truncation policy is part of prompt engineering ops. Answer: True.

Key Takeaways

Truncate with a named strategy, not ad hoc slices.
Operate in token space via tiktoken.
Protect high-priority spans.
Log losses; escalate to summarize/reject when needed.
Next: Token Optimization.

Trainer’s Guide

Lab: Implement head vs middle vs tail on a long Terms-of-Service; compare model answers to a quiz.

Discussion: When is rejecting the upload ethically required vs truncating?

Recap: Truncation is controlled forgetting under a budget. Continue with Token Optimization.