Prefix Cache — Module 12.3

How This Lesson Fits the Module & Volume

Many production prompts share a long fixed head: system instructions, tool schemas, RAG templates, or multi-turn chat history. Recomputing that shared KV on every request wastes prefill FLOPs. A prefix cache (automatic prefix caching / prompt cache) stores KV blocks for common prefixes and reuses them across requests.

This sits between speculation and batching: it reduces prefill cost so continuous batching schedulers can admit work faster.

Learning Objectives

By the end of this lesson, students should be able to:

Define prefix caching and when token prefixes are shareable.
Explain how paged KV blocks enable cross-request reuse.
Distinguish exact token-prefix match from semantic similarity.
Estimate prefill savings from a shared system prompt.
List invalidation rules (weights change, LoRA swap, tokenizer change).
Sketch a hash-keyed prefix lookup in a serving loop.

Definition

A prefix cache reuses KV-cache blocks for an identical leading token sequence across requests (or turns). On a hit, the engine skips recomputing attention for those positions and only prefills the novel suffix. Matching is exact on token IDs (and typically model identity / adapter), not approximate embedding similarity.

Where Hits Come From

System prompts

Long fixed instructions.
Highest hit rate.
Stable across users.

Multi-turn chat

History is a growing prefix.
New user turn is the suffix.
Per-session reuse.

Templates / tools

JSON schemas, few-shots.
Shared RAG wrappers.
Watch for tiny diffs.

Lookup Flow

Tokenize

Get token ID sequence

Hash prefix

Longest cached match

Attach blocks

Reference shared KV

Prefill suffix

Compute only new tokens

Condition	Cacheable?	Notes
Identical token prefix, same base model	Yes	Classic hit
Same text, different tokenizer	No	IDs diverge
Same prompt, different LoRA adapter	Usually no	Activations differ
Paraphrased system prompt	No	Not semantic cache
Weights reloaded / quantized differently	Invalidate	KV not portable

Conceptual API

from hashlib import sha256

# token_ids: list[int]; cache maps prefix hash → KV block ids
prefix_store: dict[str, list[int]] = {}

def prefix_key(token_ids: list[int], model_id: str) -> str:
    raw = model_id.encode() + b"|" + bytes(str(token_ids), "utf-8")
    return sha256(raw).hexdigest()

def attach_prefix_cache(token_ids: list[int], model_id: str):
    """Return (cached_len, block_ids) for longest cached prefix."""
    best_len, best_blocks = 0, []
    # In engines: radix/tree of blocks; here: check chunk boundaries
    for n in range(len(token_ids), 0, -1):
        key = prefix_key(token_ids[:n], model_id)
        if key in prefix_store:
            return n, prefix_store[key]
    return best_len, best_blocks

def put_prefix(token_ids: list[int], model_id: str, blocks: list[int]):
    prefix_store[prefix_key(token_ids, model_id)] = blocks

# vLLM: enable_prefix_caching=True on the engine
# OpenAI-style APIs: "prompt caching" bills discounted cached input tokens

Strengths and Tradeoffs

Strengths

Huge prefill savings on shared system prompts.
Improves TTFT (time to first token).
Natural fit with paged KV.

Tradeoffs

Extra GPU RAM for retained prefixes.
Eviction policy needed under pressure.
Brittle to tiny prompt edits.

Common Misconception

“Prefix cache is like an embedding cache for similar documents.” It requires exact token-prefix identity (plus model/adapter identity). Similar English text with different whitespace or few-shot order usually misses.

Knowledge Check

Short Answer: What does a prefix cache reuse? Answer: KV blocks for an identical leading token sequence.
True/False: Prefix matching is semantic (embedding similarity). Answer: False—exact token IDs.
Multiple Choice: Biggest wins usually come from: (a) random noise, (b) shared system prompts, (c) learning rate. Answer: (b).
Short Answer: What phase does prefix caching speed up most? Answer: Prefill / time to first token.
True/False: Changing the LoRA adapter typically invalidates cached KV. Answer: True.
Multiple Choice: On a partial hit you still: (a) recompute the entire prompt, (b) prefill only the novel suffix, (c) skip the model. Answer: (b).
Short Answer: Why do paged KV layouts help prefix sharing? Answer: Multiple requests can reference the same physical blocks.
True/False: Retained prefixes use GPU memory and need eviction. Answer: True.
Multiple Choice: Prompt caching in APIs usually discounts: (a) output tokens only, (b) cached input tokens, (c) disk rent. Answer: (b).
Short Answer: Which batching style comes next? Answer: Continuous batching.

Key Takeaways

Prefix cache = exact shared KV for common prompt heads.
Cuts prefill cost and improves TTFT in chat/RAG templates.
Depends on paged blocks and careful invalidation.
Not a semantic document cache.
Next: Continuous Batching.

Trainer’s Guide

Hands-on idea: Time TTFT for 2k-token system prompt with cold vs warm prefix cache; compute % prefill skipped.

Discussion prompt: Should product teams stabilize system prompts for cache hits even if wording is “uglier”?

Recap: Prefix caching reuses KV for shared prompt heads so servers do not re-prefill the same tokens. Continue with Continuous Batching.