← Master Index
Vol. 13 Module 13.4 Lecture

Cost Estimation

Price & Cost Control (added)

How This Lesson Fits the Module & Volume

Cost estimation forecasts spend before traffic hits production: per request, per user journey, and monthly rollups. It is how PMs and eng agree on model choices.

Combines tiktoken counts, traffic assumptions, and the price matrix from prior lessons.

Related foundations: recount tokens with tiktoken (Vol. 12) and keep payloads inside the context window (Vol. 11).

Learning Objectives

By the end of this lesson, students should be able to:

  • Build per-request and monthly estimators.
  • Use p50/p95 token distributions, not only means.
  • Include retries, tool loops, and failed calls.
  • Sensitivity-test rate and traffic changes.
  • Separate COGS estimate from retail price-to-user.
  • Document assumptions next to the spreadsheet/code.
Definition

Cost estimation is predicting USD (or credits) for LLM usage from expected tokens volumes × rates, including uncertainty and non-happy-path calls.

LayerInputsOutput
UnitTokens/request × rates$/request
JourneyCalls per task × unit$/task
FleetTasks × users × days$/month
BufferRetries, spikes, p95Contingency

Code: Monthly Sketch

import tiktoken enc = tiktoken.encoding_for_model("gpt-4o") def tokens_for(prompt: str, expected_out: int): return len(enc.encode(prompt)), expected_out IN_RATE, OUT_RATE = 2.5 / 1e6, 10.0 / 1e6 prompt_t, out_t = tokens_for("sys+user template...", 400) usd_per_req = prompt_t * IN_RATE + out_t * OUT_RATE requests_per_day = 50_000 monthly = usd_per_req * requests_per_day * 30 * 1.15 # 15% retry/tool buffer print(f"${usd_per_req:.6f}/req -> ${monthly:,.0f}/mo est.")

Point estimate

  • Fast
  • Hides tail risk
  • OK for drafts

Distributional

  • p95 aware
  • Needs traces
  • Better budgets

Scenario table

  • What-if rates/traffic
  • Extra work
  • Exec friendly

Strengths

  • Aligns model choice with finance
  • Surfaces RAG/top-k as $ knobs
  • Prevents surprise invoices

Tradeoffs

  • Garbage assumptions → garbage $
  • Price cards drift
  • Omitting agent loops underestimates
Common Misconception

“Average tokens × average price × requests is always enough.” Agent retries and p95 prompts dominate bills. Estimate with tails and failure paths.

Knowledge Check

  1. Short Answer: What three factors enter a basic estimate? Answer: Token volumes, rates, and request/traffic counts.
  2. True/False: Retries should be included in estimates. Answer: True.
  3. Multiple Choice: p95 token usage helps: (a) ignore tails, (b) plan for expensive outliers, (c) style CSS. Answer: (b).
  4. Short Answer: How does tiktoken help estimation? Answer: Measures prompt tokens for templates/samples.
  5. True/False: COGS estimate equals what you charge customers. Answer: False—retail may differ.
  6. Multiple Choice: Agent tool loops mainly: (a) reduce calls always, (b) multiply calls/tokens, (c) delete rates. Answer: (b).
  7. Short Answer: Why document assumptions? Answer: So finance/eng can refresh when traffic or prices change.
  8. Short Answer: Name a sensitivity to test. Answer: Rate change, traffic 2×, or top-k change.
  9. Multiple Choice: Monthly rollup uses: (a) only one demo call, (b) unit cost × volume, (c) font kerning. Answer: (b).
  10. True/False: Estimation is optional if you have a big credit card. Answer: False—still need control.

Key Takeaways

  • Estimate unit, journey, and monthly layers.
  • Include tails, retries, and loops.
  • Version assumptions with the price matrix.
  • Use estimators to choose models and k.
  • Next: Rate Limiting.
Trainer’s Guide

Lab: Estimate monthly COGS for 10k DAU with 3 calls/user; show p50 vs p95 token cases.

Discussion: How do you present uncertainty to executives without losing the decision?

Recap: Estimation turns tokens and rates into planned spend. Continue with Rate Limiting.