Area 05 · AI System Design · Build this

AI gateway with routing, fallback, cache and budgets

A small service that sits between your apps and model providers. Apps call one endpoint; the gateway decides which model to use, falls back on failure, serves cached answers, enforces per-user budgets and records latency and cost.

Phase 6 · Weeks 21–23

What you will have at the end

  • An OpenAI-style /v1/chat endpoint your RAG app now calls
  • Routing + fallback + semantic cache + budgets, each switchable by config
  • A dashboard of p50/p95 latency, cache hit rate and cost per user

Step by step

  1. Pass-through endpointPOST /v1/chat accepts messages and a user ID, calls one model, returns the reply, and logs model, tokens, cost and latency to Postgres.
  2. RoutingRule + classifier router between a small and a large model. Log the routing decision with every request.
  3. FallbackOrdered fallback chain on timeouts, 429s and 5xx, with a simple circuit breaker per provider.
  4. CachingExact-match cache in Redis, then a semantic cache with a tuned threshold, scoped per tenant and prompt version.
  5. BudgetsPer-user daily token budget checked before the call and recorded after. Return a clear 429 with reset time.
  6. StreamingPass through SSE streaming and record TTFT.
  7. Dashboardp50/p95 latency and TTFT, cache hit rate, fallback rate, cost per user and per model over time.
  8. Load test and write-upRun 500 requests at realistic concurrency, kill the primary provider mid-test, and document cost and latency before/after the gateway.
DONE WHEN
  • Your RAG app uses the gateway with no other change
  • A simulated provider outage causes zero failed user requests
  • The README shows cost saved by routing and caching

Stretch goals

  • Per-tenant rate limits and cost reports
  • Shadow mode: send a sample to a new model and compare offline
  • Self-hosted vLLM model as one route