Area 07 · Deployment · Concept 14 of 20

Canary Releases

Send a small share of traffic to the new version first, then roll it out to everyone.

11–20 · Production levelPhase 5 · Weeks 18–20≈ 45 min

The idea

Even with evals, real traffic finds new problems. A canary release deploys the new version alongside the old one and sends it a small share of traffic (say 5%). You watch error rate, latency, cost and quality signals (thumbs down, escalations); if they look healthy, you increase the share step by step; if not, you send all traffic back to the old version.

This matters for AI changes especially: a new prompt or model can be fine on your golden set and still behave differently on real users.

Most platforms support traffic splitting (Cloud Run revisions, Kubernetes with Argo Rollouts or a service mesh, feature flags at the app level).

Why it matters

It limits how many users see a bad release, and for how long.

Words to know

  • Traffic splitPercentage of requests sent to each version.

Try it without code

Write rollout steps

  1. Write a rollout plan: 5% → 25% → 50% → 100%.
  2. For each step, list the metrics and thresholds that must hold.

Use it in code

Canary with Cloud Run revisions.

bash
# deploy new revision without traffic
gcloud run deploy rag-api --image .../rag-api:new --no-traffic --tag canary
# send 5% to it
gcloud run services update-traffic rag-api --to-tags canary=5
# ...watch dashboards..., then promote
gcloud run services update-traffic rag-api --to-latest
# or roll back instantly
gcloud run services update-traffic rag-api --to-revisions rag-api-00041-abc=100
PRACTICE · TICK IT AFTER THIS

Release a prompt change to 10% of traffic on your deployment, compare metrics by revision, then promote or roll back.

Common mistakes

  • Canaries too small or short to show anything.
  • No per-version metrics, so you cannot compare.

Go deeper