System Design: LLM Inference Serving Platform at Scale

Name: How LLM Inference Actually Works
Uploaded: 2026-08-06T12:04:48-07:00
Duration: 40 min 41 s
Channel: ShowOffer
Description: KV cache, batching and the latency budget — the vocabulary you need before designing the serving layer.

Design the inference serving platform for a company's LLM product: millions of requests/day, a mix of short chat replies and long streaming generations, on a fixed and expensive pool of GPUs, with a p99 time-to-first-token SLO.

System Design: LLM Inference Serving Platform at Scale

What clarifying questions would you ask the interviewer?