Continuous Batching — Module 12.3

How This Lesson Fits the Module & Volume

Static batching waits for every sequence in a batch to finish before starting new work—short replies idle the GPU while long ones run. Continuous batching (iteration-level scheduling) inserts and removes requests at every decode step so the GPU stays packed with tokens that are ready right now.

Together with paged KV and prefix cache, continuous batching is the default throughput recipe in engines like vLLM. The next lecture contrasts it with classic dynamic batching.

Learning Objectives

By the end of this lesson, students should be able to:

Define continuous (iteration-level) batching.
Explain why static batches waste GPU on early finishers.
Describe the schedule loop: select, forward, emit, admit.
Relate paged KV to adding/removing requests mid-stream.
Distinguish prefill-heavy vs decode-heavy scheduling policies.
Reason about latency vs throughput tradeoffs under load.

Definition

Continuous batching is a serving strategy that rebuilds the active batch at every model iteration (token step). Finished sequences leave immediately; waiting requests can enter as soon as memory and policy allow—without waiting for the entire previous batch to complete.

Static vs Continuous

Property	Static batch	Continuous batch
Batch membership	Fixed until all done	Changes every step
Early finishers	Pad / idle slots	Slots freed immediately
New arrivals	Wait for batch end	May join next iteration
GPU utilization	Often poor under variance	High under mixed lengths
Implementation	Simple	Needs paged KV + scheduler

Scheduler Loop

1. Admit

Allocate KV; add requests

2. Select

Choose prefill/decode set

3. Forward

One iteration on GPU

4. Emit / free

Stream tokens; drop done

Illustrative Scheduler Sketch

from collections import deque

waiting = deque()   # new requests
running = []        # active sequences with KV allocated

def continuous_batch_step(engine, max_batched_tokens=4096):
    # 1) Free finished sequences from previous step
    still = []
    for req in running:
        if req.done:
            engine.free_kv(req)
        else:
            still.append(req)
    running[:] = still

    # 2) Admit waiting requests if KV pool has space
    while waiting and engine.can_allocate(waiting[0]):
        req = waiting.popleft()
        engine.alloc_kv(req)
        running.append(req)

    # 3) Build this iteration's batch (prefills + decodes)
    batch = engine.schedule(running, max_batched_tokens)
    outputs = engine.forward(batch)  # mixed prefill/decode kernels

    # 4) Append tokens; mark EOS
    for req, tok in outputs:
        req.append(tok)
        if tok == req.eos_id or req.len >= req.max_new:
            req.done = True
    return outputs

# vLLM LLMEngine / AsyncLLMEngine implement this pattern with paged attention.

Policies That Matter

FCFS

Simple fairness.
Long prompts can block.

Prefill priority

Improves TTFT.
May stall decode throughput.

Chunked prefill

Split long prefills.
Interleave with decode.

Strengths and Tradeoffs

Strengths

High tokens/sec under length variance.
Lower queueing delay for new chats.
Industry default for LLM APIs.

Tradeoffs

Complex scheduler and memory accounting.
Latency jitter under overload.
Needs careful prefill/decode mixing.

Common Misconception

“Continuous batching means infinite batch size.” The batch is still capped by KV memory and max batched tokens. Continuous only means membership updates every step—not unbounded concurrency.

Knowledge Check

Short Answer: When does continuous batching change membership? Answer: Every model iteration / decode step.
True/False: Static batching frees GPU slots as soon as one sequence finishes. Answer: False—usually waits for the whole batch.
Multiple Choice: Continuous batching relies heavily on: (a) paged KV allocation, (b) larger vocab only, (c) CSS grids. Answer: (a).
Short Answer: Name one scheduling goal besides raw throughput. Answer: TTFT / fairness / latency SLOs (any).
True/False: Finished requests can leave mid-batch in continuous scheduling. Answer: True.
Multiple Choice: Chunked prefill helps by: (a) deleting KV, (b) interleaving long prefills with decode, (c) training LoRA. Answer: (b).
Short Answer: What resource usually caps concurrent sequences? Answer: KV cache GPU memory.
True/False: Continuous batching guarantees zero latency under overload. Answer: False.
Multiple Choice: vLLM-style engines are known for: (a) continuous batching + paged attention, (b) only CPU training, (c) spreadsheet pivot. Answer: (a).
Short Answer: What related batching concept is contrasted next? Answer: Dynamic batching.

Key Takeaways

Continuous batching reschedules every token step for high utilization.
Beats static batches when output lengths vary widely.
Requires paged KV and a real admission/scheduler policy.
Trade throughput vs TTFT via prefill/decode priority.
Next: Dynamic Batching.

Trainer’s Guide

Hands-on idea: Simulate 8 requests with lengths 16–256 under static vs continuous; count idle slot-steps.

Discussion prompt: For a chatbot SLA on TTFT, when should prefill preempt decode?

Recap: Continuous batching keeps the GPU busy by admitting and retiring sequences every iteration. Continue with Dynamic Batching.