Area 05 / 08 · SYS

AI System Design

Phase 6 · Weeks 21–23

Make AI apps fast, reliable and affordable when real traffic arrives.

A prototype that works for one user can fall over, slow down or burn money at a thousand. This area covers the design decisions that keep AI systems healthy under load: routing between models, fallbacks, caching at several levels, budgets, streaming, and how model serving works on GPUs. The production half goes deep on inference optimisation, cost and multi-tenant design.

0 of 20 used in code

01–10 · The base

Learn these first.

  • 01Model RoutingEasy requests go to a cheap, fast model; hard ones go to a strong model.
  • 02Model FallbacksIf a provider fails or times out, retry on a backup model or provider.
  • 03Semantic CachingReturn a saved answer when a new question means the same thing as an old one.
  • 04Prefix CachingReuse the computation for a shared prompt start (same system prompt, same document).
  • 05Token BudgetsCap input and output tokens per request, per user or per feature.
  • 06Streaming ResponsesSend tokens to the user as they are generated, so the app feels fast.
  • 07Time to First TokenThe wait before the first word appears. This is what users feel as speed.
  • 08Throughput vs LatencyMore requests per second (big batches) vs a faster single reply. You trade one for the other.
  • 09Inference Serving with vLLMAn open-source server for running open models fast on your own GPUs.
  • 10Continuous BatchingNew requests join the running batch at every step instead of waiting for the batch to finish.

11–20 · Production level

Come back after you have shipped something.

  • 11PagedAttentionStore the KV cache in small pages, like OS memory, so far less GPU memory is wasted.
  • 12Speculative DecodingA small draft model guesses a few tokens; the big model checks them all in one pass.
  • 13GPU vs API CostWhen renting GPUs and self-hosting beats paying per token, and when it doesn’t.
  • 14Batch InferenceSend big offline jobs as a batch at a lower price when nobody is waiting for the answer.
  • 15Async Job PipelinesPut long AI tasks in a queue and run them in workers, outside the web request.
  • 16Provider Rate LimitsRequests and tokens per minute are capped. Handle it with queues, backoff and spreading load.
  • 17Multi-tenant DesignMany customers on one system, each with separate data, limits and costs.
  • 18RAG vs Fine-tuningRAG for fresh or private facts. Fine-tuning for style, format or one narrow skill.
  • 19Context Window ManagementDecide what goes in the window: trim, summarise, retrieve only what’s needed.
  • 20Cost OptimizationRouting, caching, batching, shorter prompts and smaller models, together, to cut the bill.
BUILD THIS

Put a small AI gateway in front of your app: routing between two models, fallback, semantic cache, per-user token budget and a latency + cost dashboard.

Project guide →