Cost-per-Request Optimization — Module 13.4

How This Lesson Fits the Module & Volume

This Vol. 13 capstone unifies prompting craft, token management, and price controls into one loop: minimize cost per successful request (or task), not tokens in isolation.

You will combine tiktoken measurement, budgets, caching, batch where fit, tiering, and dashboards—then hand off to Vol. 14 RAG for retrieval-heavy systems.

Related foundations: recount tokens with tiktoken (Vol. 12) and keep payloads inside the context window (Vol. 11).

Learning Objectives

By the end of this lesson, students should be able to:

Define cost-per-successful-request (CPSR) / cost-per-task.
Build an optimization backlog ranked by $ impact.
Apply 13.3 levers (count, budget, truncate, max_tokens) with 13.4 levers (cache, batch, tier).
Guardrail quality so “cheaper” does not mean “broken.”
Set a recurring review cadence with dashboards.
Prepare RAG-specific cost notes for Vol. 14.

Definition

Cost-per-request optimization is the practice of reducing expected USD (and tokens) per successful unit of work using measurement, prompt/token hygiene, pricing features, and model routing—subject to quality SLOs.

Lever (from Vol. 13)	Acts on	Capstone check
Token counting/budgets	Waste & overflow	Instrumented?
Truncation/optimization	Payload size	Eval-neutral?
max_tokens	Output tail risk	Task-tuned?
Caching / batch / tiers	$-rates & model mix	Hit rate / % routed?
Quotas / alerts / dashboards	Org control	Owners assigned?

Code: CPSR Tracker

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")

def cpsr(prompt: str, out_tokens: int, success: bool, in_rate, out_rate):
    pt = len(enc.encode(prompt))
    cost = pt * in_rate + out_tokens * out_rate
    return None if not success else cost

# Optimize E[cost | success] and success rate together — never cost alone.
rates = (2.5 / 1e6, 10.0 / 1e6)
samples = [("short prompt", 80, True), ("bloated " * 100, 80, True)]
# Compare mean CPSR before/after a change on a fixed eval suite.

Minimize tokens only

May hurt quality
Ignores rates/tiers
Incomplete

Minimize $ only

May accept failures
Hides retries
Dangerous

Minimize $/success

Aligned objective
Needs eval labels
Capstone target

Strengths

Single north-star metric for LLM COGS
Forces quality+cost co-design
Creates a repeatable review ritual

Tradeoffs

Needs reliable success labels
Cross-team coordination
Can overfit to eval suite

Common Misconception

“We cut cost 40% by deleting evals and always using the smallest model.” Unmeasured quality collapse increases retries, tickets, and churn—real CPSR goes up.

Knowledge Check

Short Answer: What is CPSR optimizing? Answer: Cost per successful request/task, not raw tokens alone.
True/False: Quality guardrails are optional in cost work. Answer: False.
Multiple Choice: Capstone combines: (a) 13.3 + 13.4 levers, (b) only CSS, (c) only Vol. 5 PCA. Answer: (a).
Short Answer: Name two 13.4 cost reducers. Answer: Prompt caching, batch discounts, or model tiering (any two).
True/False: Retries should count toward effective cost per success. Answer: True.
Multiple Choice: Next volume focus after this capstone: (a) Vol. 14 RAG, (b) Vol. 1 only, (c) printers. Answer: (a).
Short Answer: Why use tiktoken in CPSR? Answer: Measure prompt tokens consistently for experiments.
Short Answer: What dashboard signal pairs with CPSR? Answer: Success/eval rate or $/successful task.
Multiple Choice: Deleting evals to “save money”: (a) wise long-term, (b) false economy, (c) required. Answer: (b).
True/False: Vol. 13 ends by connecting token ops to dollar ops. Answer: True.

Key Takeaways

Optimize $/success with quality SLOs.
Stack 13.3 hygiene + 13.4 pricing levers.
Review CPSR on a cadence with dashboards.
Rank a backlog by measured dollar impact.
Next volume: Vol. 14 RAG—retrieval changes token economics again.

Trainer’s Guide

Lab: Run a mini workshop: pick one feature, list 5 levers, estimate $/success before/after on a 50-case eval.

Discussion: Write the one-paragraph handoff to the RAG team about pack_budget and top-k cost.

Recap: Vol. 13 closes when cheaper tokens still ship successful answers. Continue to Retrieval-Augmented Generation (RAG).