What you will have at the end
- An OpenAI-style /v1/chat endpoint your RAG app now calls
- Routing + fallback + semantic cache + budgets, each switchable by config
- A dashboard of p50/p95 latency, cache hit rate and cost per user
Step by step
- Pass-through endpointPOST /v1/chat accepts messages and a user ID, calls one model, returns the reply, and logs model, tokens, cost and latency to Postgres.
- RoutingRule + classifier router between a small and a large model. Log the routing decision with every request.
- FallbackOrdered fallback chain on timeouts, 429s and 5xx, with a simple circuit breaker per provider.
- CachingExact-match cache in Redis, then a semantic cache with a tuned threshold, scoped per tenant and prompt version.
- BudgetsPer-user daily token budget checked before the call and recorded after. Return a clear 429 with reset time.
- StreamingPass through SSE streaming and record TTFT.
- Dashboardp50/p95 latency and TTFT, cache hit rate, fallback rate, cost per user and per model over time.
- Load test and write-upRun 500 requests at realistic concurrency, kill the primary provider mid-test, and document cost and latency before/after the gateway.
DONE WHEN
- Your RAG app uses the gateway with no other change
- A simulated provider outage causes zero failed user requests
- The README shows cost saved by routing and caching
Stretch goals
- Per-tenant rate limits and cost reports
- Shadow mode: send a sample to a new model and compare offline
- Self-hosted vLLM model as one route