System Design: LLM Inference Serving Platform at Scale

Start the obvious way: a load balancer over GPU workers, one request per GPU, run the generation, return the text. A fraud classifier is one forward pass with a fixed-size output; an LLM generates token by token, autoregressively, so a request occupies its GPU for an unknown number of steps. Utilization craters while queued requests wait for whole generations to finish.

Static batching — collect N requests, run them together — is the classic fix and it fails here for the same reason: the batch completes when its longest sequence completes, so a 10-token reply waits, finished, while a 2,000-token essay grinds on. Also non-negotiable from day one: stream tokens over SSE as they decode, because the product SLO is p99 time-to-first-token, not time-to-last.

Move scheduling inside the decode loop. A continuous-batching scheduler on each worker admits new requests into the active batch at any decoding step and evicts each sequence the moment it emits its stop token — the GPU never idles waiting for a batch to drain, and short requests are no longer hostage to long ones.

This is the single highest-leverage idea in the design (it is what vLLM-class engines do), routinely worth multiple-fold throughput on real traffic. Say the batch-size trade-off out loud: deeper batches raise throughput but stretch per-step latency, so the scheduler holds a target that keeps time-to-first-token inside SLO rather than maximizing occupancy blindly.

Push concurrency up and you hit a wall that is not compute: the KV cache. Every token's attention keys and values are cached so decoding does not recompute the whole sequence per token — and that cache grows linearly with sequence length, living in GPU memory alongside weights that are already enormous (a 70B model in fp16 needs ~140 GB, forcing tensor parallelism across GPUs before request one arrives). Concurrent capacity is a memory budget, not a FLOPs budget.

Naive per-request contiguous allocation fragments that memory badly. Paged attention fixes it — the KV cache managed like OS virtual memory, in fixed-size pages — eliminating fragmentation and enabling prefix sharing: requests with a common system prompt reuse the same cached pages instead of duplicating them. A scheduler-integrated KV cache manager decides admission by whether pages are free.

Generation has two unlike phases: prefill (process the whole prompt, compute-bound, bursty) and decode (one token per step, memory-bound, steady). Mixed on one pool, a huge prompt's prefill stalls every decoding stream sharing the GPU. At sufficient scale, split them — dedicated prefill and decode pools with KV-cache pages transferred between them — so time-to-first-token and steady streaming stop fighting for the same cycles.

In front, the router becomes the policy layer: authenticate, meter tokens per tenant (accurately even for cancelled streams), route by requested model across sizes, and enforce priority admission — interactive traffic ahead of batch/offline jobs, with low-priority work queued or shed under pressure before high-priority SLOs bend. Overload must degrade by policy, not by whoever happened to arrive first.

Scaling on GPU utilization fails in a way worth narrating: utilization reads high right up until the moment latency collapses, because a saturated batch and a healthy one look identical to that gauge. The autoscaler must watch leading indicators — queue depth and projected time-to-first-token — and pre-warm capacity, since pulling tens of gigabytes of weights onto a fresh GPU takes minutes, not milliseconds.

Zero-downtime rollout follows the classic drain pattern with a twist: workers on new weights take only new requests while old workers finish their in-flight generations — some running for minutes — before retiring. Close by walking the converged lane architecture: router → continuous-batching scheduler → KV cache manager → prefill/decode pools → autoscaler, and state the two governing constraints in one breath: the GPU pool is the cost, the KV cache is the capacity.