Latency — Module 19.1

How This Lesson Fits the Module & Volume

After perplexity, evaluation leaves the loss surface and enters the product. Users do not feel nats; they feel latency—how long until the first token, how fast tokens arrive, and when the full answer lands. This lecture is the Vol. 19 metric view of inference time.

It depends on Vol. 12 inference engineering (KV cache, Flash Attention, speculative decoding, continuous / dynamic batching, prefix cache) and collides with Vol. 13 cost (cost estimation, cost per request): faster decoding often burns more GPU dollars. Vol. 18 already monitors TTFT; here you define and report the eval numbers. Next: token usage.

Learning Objectives

By the end of this lesson, students should be able to:

Define TTFT, TPOT (or TPS), and end-to-end (E2E) latency for LLM calls.
Report p50 / p95 / p99 instead of a single average.
Separate prefill (prompt) time from decode (generation) time.
Connect latency to Vol. 12 inference knobs and Vol. 13 cost trade-offs.
Design a small latency harness (streaming vs non-streaming).
Avoid apples-to-oranges compares (different max tokens, hardware, concurrency).

Definition

Latency in LLM serving is the time cost of producing a response. Practitioners split it into TTFT (time to first token: request start $\to$ first generated token), TPOT (time per output token, after the first; inverse of output tokens/sec), and E2E (request start $\to$ last token / close). Percentiles (p50, p95, p99) describe the distribution across many requests—not one lucky run.

TTFT vs TPOT vs E2E

Metric	Clock	User feels it as	Dominated by
TTFT	Start $\to$ first output token	“Did it hear me?”	Prefill, queue, network, prefix cache miss
TPOT / TPS	Inter-token gaps / tokens per second	Typing speed of the stream	Decode, batch size, speculative decoding
E2E	Start $\to$ complete	Time until they can act	TTFT + $N_{\text{out}}\times$ TPOT + trailer

Streaming (Vol. 18 SSE) makes TTFT the UX metric; non-streaming UIs only expose E2E. A long answer with great TTFT can still have terrible E2E. Conversely, a 40 ms TPOT with a 4 s queue looks “fast on paper” and awful in chat.

Percentiles, Not Averages

Mean latency hides the tail: one cold start or a huge prompt wrecks p95 while p50 looks fine. Eval reports should include at least p50 and p95, sample size, concurrency, hardware, model, max_tokens, and prompt length bucket. Load tests need a steady-state window after warmup (KV / prefix caches hot).

p50 (median)

Typical user.
Good for trend lines.
Can look healthy while p95 pages.

p95 / p99

SLO / “almost everyone.”
Catches queueing and stragglers.
Needs more samples to be stable.

Mean only

Easy to game (drop outliers).
One timeout skews everything.
Not an SLO by itself.

Prefill, Decode, and Vol. 12 Levers

Prompt tokens are processed mostly in parallel (prefill); output tokens are sequential (decode) unless speculation helps. KV cache stops re-attending the full prefix every step. Prefix cache slashes TTFT on shared system prompts. Batching raises GPU utilization and can increase per-request TTFT under load—a classic throughput vs latency trade.

Often lowers latency

KV + prefix cache hits
Speculative decoding (if accept rate is high)
Flash Attention, smaller models, fewer output tokens

Often raises it (or cost)

Huge prompts (RAG dumps) → prefill TTFT
High concurrency without enough replicas
Larger models for a tiny quality gain (Vol. 13 $)

A Minimal Latency Harness

# Illustrative streaming timers. Wire to your SDK (OpenAI / vLLM / etc.).
import time
import statistics

def percentiles(xs, ps=(50, 95)):
    xs = sorted(xs)
    out = {}
    for p in ps:
        k = min(len(xs) - 1, max(0, round((p / 100) * (len(xs) - 1))))
        out[p] = xs[k]
    return out

def time_stream(stream_fn, prompt, n_runs=30, warmup=3):
    ttfts, tpots, e2es = [], [], []
    for i in range(n_runs + warmup):
        t0 = time.perf_counter()
        t_first = None
        n_out = 0
        t_prev = None
        gaps = []
        for token in stream_fn(prompt):  # yields output tokens
            now = time.perf_counter()
            if t_first is None:
                t_first = now
            elif t_prev is not None:
                gaps.append(now - t_prev)
            t_prev = now
            n_out += 1
        t_end = time.perf_counter()
        if i < warmup:
            continue
        ttfts.append(t_first - t0)
        e2es.append(t_end - t0)
        if gaps:
            tpots.append(sum(gaps) / len(gaps))
    return {
        "ttft_s": percentiles(ttfts),
        "tpot_s": percentiles(tpots) if tpots else None,
        "e2e_s": percentiles(e2es),
        "n": n_runs,
    }

# Report alongside: model, hardware, concurrency, prompt_tokens, max_tokens
# Cost link (Vol. 13): $/1K output tokens * n_out * QPS vs GPU-hours for self-host

Related Lectures

Lecture	Role
KV cache / speculative decoding	Decode speed levers
Continuous batching	Throughput vs TTFT under load
Cost estimation / I/O pricing	Latency $\leftrightarrow$ spend
Monitoring / Streaming	TTFT in production dashboards
Token usage	Next: tokens drive both $ and E2E

Common Misconception

“Tokens per second is the only latency number that matters.” TPS ignores queueing and prefill; chat UX is often TTFT + E2E at p95. Second: comparing vendor TPS on 8-token answers to yours on 2K-token RAG dumps. Third: averaging TTFT with cold and hot prefix-cache runs without saying so. Fourth: treating timeout retries as if they were one request. Fifth: optimizing latency by silently truncating max_tokens and calling it a model win.

Knowledge Check

Short Answer: Expand TTFT, TPOT, and E2E. Answer: Time to first token; time per output token; end-to-end (start to finish).
True/False: Mean latency is sufficient for an SLO. Answer: False—use percentiles (p50/p95+).
Multiple Choice: Prefill mainly hurts: (a) TTFT on long prompts, (b) BLEU, (c) kappa. Answer: (a).
Short Answer: Why report p95 as well as p50? Answer: Tails (queues, stragglers) hide in the median.
True/False: Speculative decoding and KV cache are Vol. 12 inference topics that change TPOT/TTFT. Answer: True.
Multiple Choice: Streaming UIs make users most sensitive to: (a) TTFT, (b) wiki PPL, (c) MMLU. Answer: (a).
Short Answer: How does output length affect E2E roughly? Answer: E2E $\approx$ TTFT + $N_{\text{out}}\times$ TPOT (+ overhead).
True/False: Faster models are always cheaper (Vol. 13). Answer: False—speed can cost more GPU or premium SKUs.
Multiple Choice: Next lecture: (a) Token usage, (b) UMAP, (c) ControlNet. Answer: (a).
Short Answer: Name two confounders when comparing latency across systems. Answer: Any of: hardware, concurrency, prompt length, max tokens, warmup/cache, network, batching.

Key Takeaways

Split TTFT, TPOT, and E2E; report p50 and p95 with workload context.
Prefill vs decode explain prompt-length vs answer-length effects.
Vol. 12 inference knobs move these curves; Vol. 13 cost is the other axis.
Do not compare TPS in isolation or without warmup / token budgets.
Next: Token usage.

Trainer’s Guide

Lab: Time 30 streamed vs 30 non-streamed calls. Plot TTFT and E2E histograms. Repeat with a 200-token vs 2K-token prompt. Optionally enable prefix cache and show TTFT drop.

Whiteboard: Timeline: queue $\to$ prefill $\to$ first token $\to$ decode $\ldots$ $\to$ stop. Mark p50 vs p95. Arrow to token usage: more output tokens $\Rightarrow$ more E2E and more $.

Recap: Latency eval is TTFT + TPOT + E2E at percentiles, shaped by Vol. 12 inference and traded against Vol. 13 cost. Count what you pay for next with Token usage.

Metric	Clock	User feels it as	Dominated by
TTFT	Start \(\to\) first output token	“Did it hear me?”	Prefill, queue, network, prefix cache miss
TPOT / TPS	Inter-token gaps / tokens per second	Typing speed of the stream	Decode, batch size, speculative decoding
E2E	Start \(\to\) complete	Time until they can act	TTFT + \(N_{\text{out}}\times\) TPOT + trailer