Prompt Caching (Cost Reduction) — Module 13.4

How This Lesson Fits the Module & Volume

Stable prefixes (system prompts, tool schemas, large docs) can be cached so repeated input tokens bill at a discount. This is a major COGS lever for chat and agent apps.

Conceptually related to Vol. 12 prefix cache / KV reuse, but here the focus is billing cache hits on provider APIs.

Related foundations: recount tokens with tiktoken (Vol. 12) and keep payloads inside the context window (Vol. 11).

Learning Objectives

By the end of this lesson, students should be able to:

Explain provider prompt-cache billing vs full input rates.
Structure prompts so stable content is prefix-contiguous.
Measure cache hit rate and dollar savings.
Know TTL / invalidation behaviors at a high level.
Avoid breaking cache with volatile prefixes.
Estimate savings with tiktoken prefix lengths.

Definition

Prompt caching (cost reduction) is a provider feature that reuses prior computation/storage for repeated prompt prefixes and charges those tokens at a reduced input rate.

Do	Don’t	Why
Put stable system+tools first	Put timestamps first	Prefix must match
Keep large static docs early	Interleave user-unique noise	Breaks shared prefix
Track cache_hit tokens	Ignore usage fields	Cannot prove ROI
Version prompts deliberately	Hot-edit prod prompts hourly	Cold cache thrash

Code: Savings Estimate

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")

system = open("system.txt", encoding="utf-8").read()
prefix_tokens = len(enc.encode(system))  # assume tools+static live here too

IN, CACHED, OUT = 2.5 / 1e6, 0.25 / 1e6, 10.0 / 1e6
calls_per_day = 20_000
# Assume 90% of prefix hits cache after warmup
hit_frac = 0.9
saved = calls_per_day * prefix_tokens * hit_frac * (IN - CACHED)
print(f"prefix_tokens={prefix_tokens}, est_saved_per_day=${saved:,.2f}")

Billing prompt cache

$ discount on input
Provider-specific rules
Prefix hygiene

Inference KV/prefix cache

Latency/compute win
Serving infra
Vol. 12 deep dive

App-level response cache

Skip calls entirely
Freshness risk
Best for idempotent Qs

Strengths

Large savings on repetitive prefixes
Encourages clean prompt structure
Pairs with long tool schemas

Tradeoffs

Volatile prefixes kill hit rate
TTL/semantics vary by vendor
Savings vanish if prompts thrash

Common Misconception

“Caching means I can put a unique UUID as the first line every time.” Cache keys need a shared prefix. Unique heads force full-price input every call.

Knowledge Check

Short Answer: What gets discounted in prompt caching? Answer: Repeated input prefix tokens (per provider rules).
True/False: Putting changing timestamps at the very start helps cache hits. Answer: False.
Multiple Choice: Vol. 12 cousin concept: (a) prefix cache, (b) dropout, (c) batch norm. Answer: (a).
Short Answer: How do you prove ROI? Answer: Track cache-hit tokens and discounted rates vs baseline.
True/False: Prompt caching always skips the model call. Answer: False—it discounts repeated prefix processing/billing.
Multiple Choice: Best place for static tool schemas: (a) after user text, (b) stable early prefix, (c) random. Answer: (b).
Short Answer: Why version prompts carefully? Answer: Avoid cold-cache thrash from constant edits.
Short Answer: Which tool estimates prefix length? Answer: tiktoken (or provider counter).
Multiple Choice: App response cache: (a) may skip calls, (b) always bills full tokens, (c) trains LoRA. Answer: (a).
True/False: Cache hit rate is an ops metric. Answer: True.

Key Takeaways

Design stable prefixes for cache hits.
Measure discounted tokens and dollars.
Distinguish billing cache from KV cache.
Do not poison the prefix with volatility.
Next: Batch API Discounts.

Trainer’s Guide

Lab: Reorder a messy prompt; estimate hit savings at 20k calls/day.

Discussion: When is response caching safer than prompt-cache reliance?

Recap: Prompt caching turns stable tokens into cheaper tokens. Continue with Batch API Discounts.