Observability — Module 18.2

How This Lesson Fits the Module & Volume

Monitoring says the p95 is 8s. Observability lets you ask why this request: structured logs, distributed traces, and (for AI) prompt/version/tool spans without dumping PII into a public bucket. This is the Module 18.2 capstone: FastAPI + SDKs + Redis/Celery + stream + auth + deploy should share one trace_id.

You started Volume 18 leaving Automatic1111 studios for production SDKs. You end the backend module ready for hardware: next module opens with GPU—VRAM, utilization, and why self-host infer needs different probes than an OpenAI proxy.

Learning Objectives

By the end of this lesson, students should be able to:

Contrast metrics vs logs vs traces (three pillars) plus AI-specific spans.
Propagate trace_id through FastAPI → Celery → vendor SDK calls.
Log structured JSON (level, route, tenant, model, tokens)—redact prompts by default.
Sketch OpenTelemetry instrumentation for an LLM wrapper.
Use traces to debug TTFT vs upstream vs queue wait.
Hand off to Module 18.3 GPU metrics without confusing them with API traces.

Definition

Observability is the ability to infer internal state from external outputs: metrics (aggregates), logs (discrete events), and traces (a request’s causal DAG of spans). In LLM products, add AI telemetry: model SKU, token counts, tool names, retrieval ids, prompt template version—with privacy controls. It is not a vendor (Datadog/Honeycomb/Grafana are products). It is not evals (Vol. 19) and not GPU profiling (Module 18.3), though those feed the same ops culture.

Three Pillars + AI Spans

Signal	Answers	AI extra
Metrics	Is it bad overall?	Tokens, $, TTFT histograms
Logs	What happened at t?	Redacted prompt hash, tool errors
Traces	Where did this request spend time?	Spans: auth, cache, retrieve, llm, tools

Trace + Structured Log Sketch

# pip install fastapi uvicorn openai opentelemetry-api opentelemetry-sdk
import hashlib, json, logging, os, time
from fastapi import FastAPI, Request
from openai import OpenAI
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor

logging.basicConfig(level=os.getenv("LOG_LEVEL", "INFO"))
log = logging.getLogger("chat")
trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
tracer = trace.get_tracer("chat-api")
app = FastAPI()
client = OpenAI()

@app.middleware("http")
async def request_id_mw(request: Request, call_next):
    rid = request.headers.get("x-request-id") or hashlib.sha1(str(time.time()).encode()).hexdigest()[:12]
    request.state.request_id = rid
    with tracer.start_as_current_span("http.request") as span:
        span.set_attribute("http.route", request.url.path)
        span.set_attribute("request.id", rid)
        resp = await call_next(request)
        resp.headers["x-request-id"] = rid
        span.set_attribute("http.status_code", resp.status_code)
        return resp

@app.post("/v1/chat")
def chat(request: Request, body: dict):
    rid = getattr(request.state, "request_id", "-")
    with tracer.start_as_current_span("llm.chat") as span:
        span.set_attribute("llm.model", "gpt-4.1-mini")
        span.set_attribute("llm.provider", "openai")
        t0 = time.perf_counter()
        resp = client.chat.completions.create(
            model="gpt-4.1-mini",
            messages=[{"role": "user", "content": body["prompt"]}],
            max_tokens=256,
        )
        u = resp.usage
        span.set_attribute("llm.tokens.input", u.prompt_tokens if u else 0)
        span.set_attribute("llm.tokens.output", u.completion_tokens if u else 0)
        log.info(json.dumps({
            "msg": "chat_ok",
            "request_id": rid,
            "model": resp.model,
            "ms": int((time.perf_counter() - t0) * 1000),
            "input_tokens": u.prompt_tokens if u else None,
            "prompt_sha": hashlib.sha256(body["prompt"].encode()).hexdigest()[:16],
            # do NOT log raw prompt/PII by default
        }))
        return {"text": resp.choices[0].message.content, "request_id": rid}

# Celery: pass request_id in task kwargs; continue the trace with a link/span
# GPU next module: emit nvml util as metrics; still correlate via request_id if infer is local

Privacy and Tooling

Redact by default

Prompt hash, not prompt
Tenant id, not email body
Allow-list fields in logs

When you keep prompts

Separate store, access control
Retention + deletion SLA
Needed for evals (Vol. 19)—not Grafana

Stack (examples)

OTel → Tempo/Jaeger/Honeycomb
Loki/ELK for JSON logs
Prometheus already from Monitoring

Observability wins

Debug one slow chat across Pods
See cache hit vs LLM vs tools
Correlate deploy SHA to traces

Failure modes

PII in traces = incident
Trace everything, sample nothing = cost bomb
No request_id on Celery = blind workers

Related Lectures

Lecture	Role
Monitoring	Aggregates; this page is causal detail
FastAPI / Celery	Span boundaries
OpenAI / Anthropic / Gemini	Provider spans
GPU	Next module—hardware signals
Vol. 19 Evaluation	Quality, not just latency

Common Misconception

“We log every prompt to stdout, so we are observable.” That is a privacy incident with extra steps. Second: metrics dashboards without trace_id cannot explain one VIP outage. Third: OpenTelemetry is not automatic understanding—you still name spans. Fourth: GPU util in nvidia-smi is not a substitute for API traces (and vice versa). Fifth: finishing 18.2 does not mean you self-host 70B; that decision is Module 18.3 / 18.4.

Knowledge Check

Short Answer: Name the three classic observability pillars. Answer: Metrics, logs, and traces.
True/False: Observability is the same as monitoring. Answer: False—monitoring is mostly metrics/alerts; observability includes causal traces/logs.
Multiple Choice: A span attribute you should usually include: (a) llm.model, (b) raw SSN, (c) CFG sampler. Answer: (a).
Short Answer: Why pass request_id into Celery tasks? Answer: To continue/link the trace across web and worker processes.
True/False: Log full user prompts by default. Answer: False—hash/redact; store prompts only under a privacy policy.
Multiple Choice: TTFT vs total latency is easiest to see with: (a) traces/spans, (b) only docker ps, (c) LoRA. Answer: (a).
Short Answer: What module comes immediately after this lecture? Answer: Module 18.3, starting with GPU.
True/False: OpenTelemetry is a specific SaaS vendor. Answer: False—it is a standard/SDK; backends vary.
Multiple Choice: Volume 18 began (after Vol. 17 UIs) with: (a) OpenAI SDK, (b) DBSCAN, (c) Firefly. Answer: (a).
Short Answer: Name one AI-specific span besides the HTTP request. Answer: Any of: llm.chat, cache lookup, retrieve/RAG, tool call, stream TTFT.

Key Takeaways

Observability = metrics + logs + traces (+ careful AI telemetry).
One request_id / trace across FastAPI, Redis/Celery, and SDKs.
Redact prompts; sample traces; name spans honestly.
Module 18.2 complete: you can ship and debug an AI backend.
Next module: GPU (Hardware & Model Optimization).

Trainer’s Guide

Lab (capstone 18.2): Take the FastAPI+Redis+optional Celery app. Add request_id middleware, JSON logs, and console OTel spans. Break upstream OpenAI (bad key) and find the failure from a single x-request-id. Discuss what would change if infer moved onto a GPU worker.

Whiteboard: Full Vol. 18 so far: SDKs → API → FastAPI → Docker/K8s → Redis/Celery → WS/SSE → Auth → Deploy → Monitor → Observe. Arrow out to GPU.

Recap: Observability explains each request across the backend. Hardware next: GPU.