The idea
Static batching groups requests, runs them together until all finish, then starts the next group. Short requests finish early and their slots sit idle while the longest one completes; new requests wait.
Continuous (in-flight) batching schedules at the level of each decoding step. When a sequence finishes, its slot is freed immediately and a waiting request is added on the next step. The GPU stays busy and queues stay short.
Combined with PagedAttention, this is why modern servers reach far higher throughput than naive serving at similar latency.
It is the scheduling idea behind efficient LLM serving and a common system design interview topic.
Picture it
Words to know
- Iteration-level schedulingDeciding batch membership at every decode step.
Try it without code
Simulate on paper
- Draw 4 slots and 6 requests of different lengths.
- Schedule them static and continuous.
- Count idle slot-steps in each.
Use it in code
A toy simulator comparing idle slots.
def simulate(lengths, slots, continuous):
queue, active, t, idle = list(lengths), [], 0, 0
while queue or active:
if continuous or not active:
while queue and len(active) < slots:
active.append(queue.pop(0))
idle += slots - len(active)
active = [n - 1 for n in active]
active = [n for n in active if n > 0]
t += 1
return t, idle
lengths = [8, 2, 3, 7, 2, 5, 1, 6]
print("static ", simulate(lengths, 4, False))
print("continuous", simulate(lengths, 4, True))Extend the simulator to report average wait time per request for both strategies.
Common mistakes
- Confusing batch API pricing (offline jobs) with continuous batching (a server scheduling technique).