Area 05 · AI System Design · Concept 10 of 20

Continuous Batching

New requests join the running batch at every step instead of waiting for the batch to finish.

01–10 · The basePhase 6 · Weeks 21–23≈ 45 min

The idea

Static batching groups requests, runs them together until all finish, then starts the next group. Short requests finish early and their slots sit idle while the longest one completes; new requests wait.

Continuous (in-flight) batching schedules at the level of each decoding step. When a sequence finishes, its slot is freed immediately and a waiting request is added on the next step. The GPU stays busy and queues stay short.

Combined with PagedAttention, this is why modern servers reach far higher throughput than naive serving at similar latency.

Why it matters

It is the scheduling idea behind efficient LLM serving and a common system design interview topic.

Picture it

Words to know

  • Iteration-level schedulingDeciding batch membership at every decode step.

Try it without code

Simulate on paper

  1. Draw 4 slots and 6 requests of different lengths.
  2. Schedule them static and continuous.
  3. Count idle slot-steps in each.

Use it in code

A toy simulator comparing idle slots.

python
def simulate(lengths, slots, continuous):
    queue, active, t, idle = list(lengths), [], 0, 0
    while queue or active:
        if continuous or not active:
            while queue and len(active) < slots:
                active.append(queue.pop(0))
        idle += slots - len(active)
        active = [n - 1 for n in active]
        active = [n for n in active if n > 0]
        t += 1
    return t, idle

lengths = [8, 2, 3, 7, 2, 5, 1, 6]
print("static    ", simulate(lengths, 4, False))
print("continuous", simulate(lengths, 4, True))
PRACTICE · TICK IT AFTER THIS

Extend the simulator to report average wait time per request for both strategies.

Common mistakes

  • Confusing batch API pricing (offline jobs) with continuous batching (a server scheduling technique).

Go deeper