The idea
Even with evals, real traffic finds new problems. A canary release deploys the new version alongside the old one and sends it a small share of traffic (say 5%). You watch error rate, latency, cost and quality signals (thumbs down, escalations); if they look healthy, you increase the share step by step; if not, you send all traffic back to the old version.
This matters for AI changes especially: a new prompt or model can be fine on your golden set and still behave differently on real users.
Most platforms support traffic splitting (Cloud Run revisions, Kubernetes with Argo Rollouts or a service mesh, feature flags at the app level).
It limits how many users see a bad release, and for how long.
Words to know
- Traffic splitPercentage of requests sent to each version.
Try it without code
Write rollout steps
- Write a rollout plan: 5% → 25% → 50% → 100%.
- For each step, list the metrics and thresholds that must hold.
Use it in code
Canary with Cloud Run revisions.
# deploy new revision without traffic
gcloud run deploy rag-api --image .../rag-api:new --no-traffic --tag canary
# send 5% to it
gcloud run services update-traffic rag-api --to-tags canary=5
# ...watch dashboards..., then promote
gcloud run services update-traffic rag-api --to-latest
# or roll back instantly
gcloud run services update-traffic rag-api --to-revisions rag-api-00041-abc=100Release a prompt change to 10% of traffic on your deployment, compare metrics by revision, then promote or roll back.
Common mistakes
- Canaries too small or short to show anything.
- No per-version metrics, so you cannot compare.