Dynamic Batching — Module 12.3

How This Lesson Fits the Module & Volume

Continuous batching updates membership every token. Dynamic batching (classic request-level / micro-batching) waits a short window to gather arrivals, then runs them together as a fixed batch until completion—widely used in Triton, TensorFlow Serving, and encoder or embedding services.

Closing Module 12.3, compare both strategies so you can pick the right tool: iteration-level for LLM decode, windowed dynamic batches for uniform one-shot models. Module 12.4 then shifts from serving to fine-tuning.

Learning Objectives

By the end of this lesson, students should be able to:

Define dynamic (windowed) batching and its preferred max-batch / max-delay knobs.
Contrast dynamic vs continuous batching for LLM decode.
Explain padding and shape bucketing for variable-length inputs.
Choose batching style for embeddings, classifiers, and chat LLMs.
Sketch a max-wait micro-batcher in Python.
Discuss latency SLOs versus throughput under bursty traffic.

Definition

Dynamic batching aggregates concurrent inference requests that arrive within a time window (or until a size cap), then executes them as one batch. Membership is decided before the forward pass and typically stays fixed for that execution—unlike continuous batching, which reshuffles every decode step.

Continuous vs Dynamic (Serving Throughput)

Dimension	Dynamic batching	Continuous batching
Decision point	Request / window	Every token iteration
Batch lifetime	One (or few) forwards	Many decode steps
Best for	Encoders, embeddings, classifiers	Autoregressive LLMs
Early finish	Does not free mid-generation	Frees immediately
Knobs	max_batch, max_delay_ms	max_batched_tokens, KV budget

Windowed Micro-Batcher

Arrive

Requests enter queue

Wait

Until size or timeout

Pad / bucket

Align shapes

Forward

Single batched call

import asyncio
import time

class DynamicBatcher:
    def __init__(self, infer_fn, max_batch=32, max_delay_ms=10):
        self.infer_fn = infer_fn
        self.max_batch = max_batch
        self.max_delay = max_delay_ms / 1000.0
        self.queue = []
        self.lock = asyncio.Lock()

    async def submit(self, item):
        fut = asyncio.get_event_loop().create_future()
        async with self.lock:
            self.queue.append((item, fut))
            if len(self.queue) == 1:
                asyncio.create_task(self._drain())
        return await fut

    async def _drain(self):
        await asyncio.sleep(self.max_delay)
        async with self.lock:
            batch = self.queue[: self.max_batch]
            self.queue = self.queue[self.max_batch :]
            if self.queue:
                asyncio.create_task(self._drain())
        items = [x for x, _ in batch]
        outs = self.infer_fn(items)  # e.g., embedder or classifier
        for (_, fut), out in zip(batch, outs):
            fut.set_result(out)

# Triton Inference Server: dynamic_batching { max_queue_delay_microseconds: ... }

When to Use Which

Prefer dynamic

One-shot models (Bi-Encoder).
Uniform latency targets.
Simple Triton configs.

Prefer continuous

Chat / long generation.
High length variance.
KV-heavy LLM serving.

Hybrid reality

Gateway may dynamically batch.
LLM engine batches continuously.
Both appear in one stack.

Strengths and Tradeoffs

Strengths

Simple, mature in classic ML serving.
Amortizes kernel launch / weight reads.
Easy max-delay latency control.

Tradeoffs

Poor fit for multi-step LLM decode alone.
Padding waste on jagged lengths.
Waiting adds latency even under light load.

Common Misconception

“Dynamic and continuous batching are the same buzzword.” Dynamic = gather requests, then run. Continuous = repack every token. LLM APIs need the latter; embedding microservices often need the former.

Knowledge Check

Short Answer: Name the two classic knobs of dynamic batching. Answer: Max batch size and max queue delay.
True/False: Dynamic batching typically reshuffles membership every decode token. Answer: False—that is continuous batching.
Multiple Choice: Dynamic batching fits best: (a) long chat decode, (b) one-shot encoders/embeddings, (c) only CSS. Answer: (b).
Short Answer: Why does waiting hurt under light load? Answer: Requests sit idle until the delay timer fires.
True/False: Padding variable-length sequences can waste compute in a dynamic batch. Answer: True.
Multiple Choice: Continuous batching frees slots: (a) only at midnight, (b) when a sequence finishes mid-generation, (c) never. Answer: (b).
Short Answer: Where is dynamic batching commonly configured? Answer: Triton / classic model servers (or similar).
True/False: A full stack may use both styles in different tiers. Answer: True.
Multiple Choice: Module 12.3 focused on: (a) inference optimization, (b) only CNNs, (c) SQL joins. Answer: (a).
Short Answer: What module topic follows (12.4 opener)? Answer: Full fine-tuning.

Key Takeaways

Dynamic batching windows requests into fixed execution batches.
Excellent for one-shot models; incomplete alone for LLM decode.
Continuous batching is the LLM-serving counterpart.
Pick based on workload shape, not buzzwords.
Next module: Full Fine Tuning.

Trainer’s Guide

Hands-on idea: Plot latency vs max_delay for an embedding service at fixed QPS; find the knee.

Discussion prompt: For RAG, which batcher sits in front of the bi-encoder vs the generator LLM?

Recap: Dynamic batching gathers requests by size and delay; continuous batching keeps LLM decode packed. Proceed to Full Fine Tuning.