Design a chat-completion service and the scarce resource is not CPU or database connections - it is accelerator memory and bandwidth. KV-cache sizing, prefill versus decode, continuous batching, TTFT and TPOT, and token-based admission control, worked with numbers.
The initial design is rarely what separates a hire from a no-hire in an AI system design loop — it is how you handle the follow-up. Here are the five questions that come up constantly across RAG, agent orchestration, and inference serving rounds, with what a strong answer actually sounds like.
RAG pipelines, agent orchestration, and LLM inference serving have become the new bar for senior AI engineering interviews. Here is how to prepare for all three without wasting weeks on the wrong material.