Monitoring — Module 18.2

How This Lesson Fits the Module & Volume

After deployment, silence is not success. Monitoring watches golden signals for the AI API: latency (including TTFT for streams), error rates, saturation, and—unique to this volume—token usage and estimated cost per model/tenant. You already return usage from Module 18.1 SDKs; now scrape or emit it.

The next lecture, observability, widens the lens (traces, structured logs, correlation). Monitoring is the dashboard and the pager. Then Module 18.3 starts with GPU hardware signals.

Learning Objectives

By the end of this lesson, students should be able to:

List golden signals for an LLM proxy (not only CPU).
Emit Prometheus-style metrics from FastAPI (QPS, latency, tokens, 429s).
Alert on error budget, upstream 502s, and cost spikes—not on vanity charts.
Track cache hit ratio and queue depth (Redis/Celery).
Separate product SLOs from vendor SLA you do not control.
Hand off to traces/logs in the observability lecture.

Definition

Monitoring is the practice of collecting time-series metrics (and simple health checks), visualizing them, and alerting when they leave SLO bounds. For AI backends, metrics include HTTP RED/USE plus tokens in/out, estimated USD, TTFT, stream aborts, and provider breakdown. Monitoring asks “is it broken / expensive / slow right now?” Observability asks “why, for this request?”

What to Measure

Signal	Example metric	Why
Rate	`http_requests_total`	Load / attacks
Errors	5xx, 429, upstream 502	SLO + wallet protection
Duration	p50/p95 latency, TTFT	Chat UX
Tokens	`llm_tokens_total{io,model}`	Cost
Cache	Redis hit ratio	Did caching help?
Queue	Celery depth / time-to-start	Job UX
Health	`/health` probe success	K8s + deploy smoke

Prometheus Metrics Sketch

# pip install fastapi uvicorn prometheus-client openai
import time
from fastapi import FastAPI, Response
from openai import OpenAI
from prometheus_client import Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST

app = FastAPI()
client = OpenAI()

REQS = Counter("http_requests_total", "HTTP requests", ["route", "code"])
LAT = Histogram("http_request_duration_seconds", "Latency", ["route"])
TTFT = Histogram("llm_ttft_seconds", "Time to first token")
TOKS = Counter("llm_tokens_total", "LLM tokens", ["model", "io"])
COST = Counter("llm_cost_usd_total", "Estimated USD", ["model"])

# toy rates — keep a real price table in config, not hardcoded forever
PRICE = {"gpt-4.1-mini": {"in": 0.4 / 1e6, "out": 1.6 / 1e6}}

@app.get("/metrics")
def metrics():
    return Response(generate_latest(), media_type=CONTENT_TYPE_LATEST)

@app.post("/v1/chat")
def chat(body: dict):
    route = "/v1/chat"
    t0 = time.perf_counter()
    try:
        resp = client.chat.completions.create(
            model="gpt-4.1-mini",
            messages=[{"role": "user", "content": body["prompt"]}],
            max_tokens=256,
        )
        REQS.labels(route, "200").inc()
    except Exception:
        REQS.labels(route, "502").inc()
        LAT.labels(route).observe(time.perf_counter() - t0)
        raise
    LAT.labels(route).observe(time.perf_counter() - t0)
    u = resp.usage
    if u:
        TOKS.labels(resp.model, "in").inc(u.prompt_tokens)
        TOKS.labels(resp.model, "out").inc(u.completion_tokens)
        p = PRICE.get("gpt-4.1-mini", {"in": 0, "out": 0})
        COST.labels(resp.model).inc(u.prompt_tokens * p["in"] + u.completion_tokens * p["out"])
    return {"text": resp.choices[0].message.content}

# Alert ideas: rate(http_requests_total{code=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
#             increase(llm_cost_usd_total[1h]) > budget
# TTFT: start timer before first SSE delta (Streaming lecture)

SLOs vs Vendor Luck

You own

Auth failures, your 5xx
Queue depth, cache, saturation
Cost caps per tenant

Vendor owns

OpenAI/Anthropic/Gemini outages
Their rate limits (you still 429)
Model quality (Vol. 19 evals)

Alert hygiene

Page on SLO burn, not CPU 60%
Staging vs prod budgets
No pages without a runbook

Monitoring buys

Pagers before Twitter
Cost visibility per SKU
Deploy confidence (dashboards)

Not enough alone

Averages hide one tenant melting you
No trace = cannot debug one chat
Quality ≠ latency (Vol. 19)

Related Lectures

Lecture	Role
Deployment	Smoke + dashboards after rollout
Observability	Traces/logs next
Redis / Celery	Hit ratio + queue metrics
OpenAI SDK	Source of usage fields
GPU	Util / VRAM when you self-host

Common Misconception

“Uptime 99.9% means the AI product works.” You can serve 200s of empty or hallucinated text. Monitoring latency/cost ≠ eval quality (Vol. 19). Second: only watching CPU on a chat proxy that waits on OpenAI. Third: alerting on every 429—sometimes 429 is the system working. Fourth: putting API keys in Grafana screenshots.

Knowledge Check

Short Answer: Name three golden signals for an LLM API. Answer: Any of: rate, errors, latency/TTFT, tokens/cost, saturation, queue depth, cache hit.
True/False: Token usage is optional for production monitoring. Answer: False—it is the cost signal.
Multiple Choice: TTFT means: (a) time to first token, (b) t-SNE, (c) TLS. Answer: (a).
Short Answer: Why label metrics by model? Answer: SKUs differ in price and latency; you need breakdowns.
True/False: Monitoring replaces distributed traces. Answer: False—observability lecture covers why.
Multiple Choice: A useful alert is: (a) SLO error-budget burn or cost spike, (b) CPU exactly 12%, (c) CFG 7. Answer: (a).
Short Answer: Where do token counts come from? Answer: Vendor SDK usage fields (or your tokenizer estimate).
True/False: Vendor outages are still your user’s outage. Answer: True—you must detect/communicate even if you do not control them.
Multiple Choice: Next lecture: (a) Observability, (b) Naive Bayes, (c) DreamBooth. Answer: (a).
Short Answer: Why track Celery queue depth? Answer: Jobs can look “up” while wait time explodes.

Key Takeaways

Monitor RED + tokens/cost + TTFT + queues/cache.
Prometheus (or equivalent) from FastAPI; alert on SLOs and spend.
You do not control vendor SLA; you still own the user experience.
Quality evals are Vol. 19; metrics ≠ truthfulness.
Next: Observability.

Trainer’s Guide

Lab: Scrape /metrics with Prometheus (or even curl before/after 20 chats). Graph tokens and p95. Inject a 502 and watch the error counter. Add a fake budget alert.

Whiteboard: Dashboard tiles: QPS, p95, 5xx, $, TTFT, queue. Arrow “why is this one request slow?” to Observability.

Recap: Monitoring tells you if the API is slow, broken, or expensive. Explain a single request next with Observability.