Area 05 / 08 · SYS
AI System Design
Phase 6 · Weeks 21–23Make AI apps fast, reliable and affordable when real traffic arrives.
A prototype that works for one user can fall over, slow down or burn money at a thousand. This area covers the design decisions that keep AI systems healthy under load: routing between models, fallbacks, caching at several levels, budgets, streaming, and how model serving works on GPUs. The production half goes deep on inference optimisation, cost and multi-tenant design.
0 of 20 used in code
01–10 · The base
Learn these first.
- 01Model RoutingEasy requests go to a cheap, fast model; hard ones go to a strong model.
- 02Model FallbacksIf a provider fails or times out, retry on a backup model or provider.
- 03Semantic CachingReturn a saved answer when a new question means the same thing as an old one.
- 04Prefix CachingReuse the computation for a shared prompt start (same system prompt, same document).
- 05Token BudgetsCap input and output tokens per request, per user or per feature.
- 06Streaming ResponsesSend tokens to the user as they are generated, so the app feels fast.
- 07Time to First TokenThe wait before the first word appears. This is what users feel as speed.
- 08Throughput vs LatencyMore requests per second (big batches) vs a faster single reply. You trade one for the other.
- 09Inference Serving with vLLMAn open-source server for running open models fast on your own GPUs.
- 10Continuous BatchingNew requests join the running batch at every step instead of waiting for the batch to finish.
11–20 · Production level
Come back after you have shipped something.
- 11PagedAttentionStore the KV cache in small pages, like OS memory, so far less GPU memory is wasted.
- 12Speculative DecodingA small draft model guesses a few tokens; the big model checks them all in one pass.
- 13GPU vs API CostWhen renting GPUs and self-hosting beats paying per token, and when it doesn’t.
- 14Batch InferenceSend big offline jobs as a batch at a lower price when nobody is waiting for the answer.
- 15Async Job PipelinesPut long AI tasks in a queue and run them in workers, outside the web request.
- 16Provider Rate LimitsRequests and tokens per minute are capped. Handle it with queues, backoff and spreading load.
- 17Multi-tenant DesignMany customers on one system, each with separate data, limits and costs.
- 18RAG vs Fine-tuningRAG for fresh or private facts. Fine-tuning for style, format or one narrow skill.
- 19Context Window ManagementDecide what goes in the window: trim, summarise, retrieve only what’s needed.
- 20Cost OptimizationRouting, caching, batching, shorter prompts and smaller models, together, to cut the bill.
BUILD THIS
Put a small AI gateway in front of your app: routing between two models, fallback, semantic cache, per-user token budget and a latency + cost dashboard.
Project guide →