The question behind the question
'Design a service that serves a large language model to millions of users' is now a standard system design prompt at AI labs and at companies that run their own models. Candidates who have prepared for conventional design rounds reach for load balancers, caches and sharded databases. Those still appear, but they are not where the interesting decisions are.
The interviewer wants to see that you know what the bottleneck is. In LLM serving it is the accelerator: its memory capacity, which bounds how many conversations fit at once, and its memory bandwidth, which bounds how fast each one generates. Everything else in the design - batching, scheduling, routing, rate limiting - is a way of using that scarce resource well. Show that with arithmetic and the rest of the round becomes a conversation about trade-offs rather than a recitation of components.
The numbers in this post use a hypothetical 70-billion-parameter model with 80 layers, 8 key-value heads of dimension 128, and 16-bit weights and cache. They are deliberately round. In an interview, state the assumptions out loud and the interviewer will correct any that matter.
Memory: weights plus KV cache
Weights first. Seventy billion parameters at two bytes each is 140 GB, which does not fit on a single 80 GB accelerator, so the model is split across several - tensor parallelism across, say, four devices gives 320 GB of memory, of which the weights use 140.
The rest goes to the KV cache, and this is the number that sets concurrency. Per token, the cache stores keys and values for every layer: 2 x 80 layers x 8 heads x 128 dimensions x 2 bytes, about 320 KB. A conversation with 4,000 tokens of context therefore costs about 1.3 GB. With roughly 150 GB free after weights and activation overhead, that is about a hundred and ten concurrent 4k-token sequences per four-device replica - and far fewer at long context.
Now the design follows. Concurrency per replica is a memory question, so it falls as context length rises. That is why long-context requests are often priced and scheduled differently, and why prefix caching - reusing the cache for a shared system prompt across requests - is worth real money.
layers, kv_heads, head_dim, bytes_ = 80, 8, 128, 2
params, devices, mem_per_device = 70e9, 4, 80e9
kv_per_token = 2 * layers * kv_heads * head_dim * bytes_ # keys + values
weights = params * bytes_
free = devices * mem_per_device - weights - 0.1 * devices * mem_per_device
print(f"KV cache / token {kv_per_token/1e3:.0f} KB") # 328 KB
print(f"4k-token sequence {4096*kv_per_token/1e9:.2f} GB") # 1.34 GB
print(f"concurrent 4k seqs {free // (4096*kv_per_token):.0f}") # ~110
print(f"concurrent 32k seqs {free // (32768*kv_per_token):.0f}") # ~13Speed: prefill is compute, decode is bandwidth
A request has two phases. Prefill processes the whole prompt in parallel: lots of arithmetic on lots of tokens at once, so it is compute-bound and fast per token. It determines time to first token, TTFT. Decode then generates one token at a time, and every step must read all the weights from memory to produce a single token per sequence. It is bandwidth-bound, and it determines time per output token, TPOT.
The decode floor is easy to estimate. Reading 140 GB of weights across four devices at roughly 3 TB/s each takes about 12 milliseconds, so one sequence decoding alone gets at best around 80 tokens per second. The crucial point is that a batch of 64 sequences costs almost the same weight read as a batch of one. Batching is how you turn a bandwidth-bound process into high throughput - until the KV cache, which grows with every sequence, runs out of room.
So the core trade-off of the whole design is a single dial: bigger batches give more tokens per second per dollar and slower tokens per user. Your latency targets, expressed as TTFT and TPOT percentiles, decide where you set it.
Batching, scheduling and streaming
Static batching - wait for N requests, run them together to completion - wastes capacity, because sequences finish at different lengths and the batch is only as fast as its longest member. Continuous batching schedules at the level of individual decode steps: a finished sequence leaves and a waiting one joins on the next iteration. Paired with a paged KV cache that allocates memory in blocks rather than one contiguous slab per sequence, as popularised by vLLM, it is the default in modern serving stacks.
Prefill and decode interfere with each other: a long prompt arriving mid-batch stalls everyone's next token. Common mitigations are chunked prefill, which splits long prompts into pieces interleaved with decode steps, and disaggregation, which runs prefill and decode on separate pools. Mention both and say what each costs - chunking adds scheduling complexity; disaggregation means moving KV cache between machines.
Tokens are streamed to the client as they are produced, typically with server-sent events, because perceived latency is dominated by the first token. That also means the API layer holds many long-lived connections, which shapes the choice of gateway.
Admission control in tokens, not requests
A request asking for 50 output tokens and one asking for 4,000 differ in cost by almost two orders of magnitude, so rate limits and admission control are expressed in tokens. A sound design estimates each request's cost from prompt length plus max_tokens, admits it only if the replica has KV-cache headroom, queues it otherwise, and sheds load with a clear 429 and a retry-after when the queue exceeds its latency budget.
Close the round by naming what you would measure: TTFT and TPOT at p50 and p99, tokens per second per device, KV-cache utilisation, queue depth and the rejection rate. Then say which one you would alert on. An interviewer who hears 'p99 TTFT, because queueing shows up there first' knows you have thought about running this system, not just drawing it.