← Master Index
Vol. 13 Module 13.4 Lecture

Cost Monitoring Dashboards

Price & Cost Control (added)

How This Lesson Fits the Module & Volume

Without dashboards, token and dollar controls are vibes. Ops needs live views of usage, cost, cache hits, tier mix, errors, and budget burn.

Capstone-adjacent observability for everything taught in 13.3–13.4.

Related foundations: recount tokens with tiktoken (Vol. 12) and keep payloads inside the context window (Vol. 11).

Learning Objectives

By the end of this lesson, students should be able to:

  • List essential LLM cost dashboard panels.
  • Break down cost by model, feature, tenant, and env.
  • Chart token totals with input/output split.
  • Overlay budgets and alert thresholds.
  • Track quality alongside cost (not cost alone).
  • Automate daily digests for stakeholders.
Definition

Cost monitoring dashboards are operational views that visualize LLM token usage, spend, quotas, and related efficiency metrics (cache hit rate, tier mix, error/retry rates) over time.

PanelMetricsAction link
Spend burn$/hour, $/day vs budgetAlerts lesson
Token splitIn vs out tokensPricing tiers
Model mix% small/largeTiering
EfficiencyCache hit %, $/successful taskCaching / optimization
Reliability429s, overflows, retriesRate limit / overflow

Code: Emit Cost Metrics

def record_usage(metrics, *, model, feature, tenant, prompt_t, out_t, rates): cost = prompt_t * rates["in"] + out_t * rates["out"] metrics.incr("llm_tokens_in", prompt_t, tags=dict(model=model, feature=feature, tenant=tenant)) metrics.incr("llm_tokens_out", out_t, tags=dict(model=model, feature=feature, tenant=tenant)) metrics.incr("llm_cost_usd", cost, tags=dict(model=model, feature=feature, tenant=tenant)) return cost # Dashboard queries group by tags over time; alert on sum(cost) vs budget.

Finance invoice

  • Authoritative late
  • Poor debug
  • Monthly

Live usage metrics

  • Actionable
  • Needs instrumentation
  • Minute/hour

Joined view

  • Cost + quality
  • Harder pipelines
  • Best for orgs

Strengths

  • Makes COGS visible daily
  • Enables attribution by feature
  • Supports executive trust

Tradeoffs

  • Bad tags → useless charts
  • Cardinality explosions
  • Cost without quality misleads
Common Misconception

“We graph total tokens only.” Without input/output split, model, and feature tags, you cannot tell whether RAG or chatter caused the spike.

Knowledge Check

  1. Short Answer: Name three dashboard dimensions. Answer: Model, feature, tenant/env (or in/out, budget—any three).
  2. True/False: Invoices alone are enough for realtime ops. Answer: False.
  3. Multiple Choice: Cache hit % belongs under: (a) efficiency panels, (b) CSS, (c) dropout. Answer: (a).
  4. Short Answer: Why tag by feature? Answer: Attribute spend to product surfaces.
  5. True/False: Cost dashboards should ignore 429/overflow rates. Answer: False.
  6. Multiple Choice: Input/output split panels support: (a) tiered pricing analysis, (b) font choice, (c) PCA. Answer: (a).
  7. Short Answer: What joins cost to quality? Answer: $/successful task or eval pass rate overlays.
  8. Short Answer: Where do token numbers originate? Answer: API usage (+ local tiktoken estimates).
  9. Multiple Choice: High cardinality tags risk: (a) useless/expensive metrics, (b) free GPUs, (c) better BLEU always. Answer: (a).
  10. True/False: Daily digests help non-oncall stakeholders. Answer: True.

Key Takeaways

  • Instrument cost with useful tags.
  • Split tokens; overlay budgets.
  • Watch efficiency and reliability together.
  • Prefer live metrics over waiting on invoices.
  • Next: Cost-per-Request Optimization (Vol. 13 capstone).
Trainer’s Guide

Lab: Sketch a 6-panel LLM cost dashboard for a multi-tenant SaaS.

Discussion: Which single panel would you show a CEO weekly?

Recap: Dashboards make token money observable. Continue with the Vol. 13 capstone: Cost-per-Request Optimization.