Area 05 · AI System Design · Concept 02 of 20

Model Fallbacks

If a provider fails or times out, retry on a backup model or provider.

01–10 · The basePhase 6 · Weeks 21–23≈ 45 min

The idea

Model APIs have outages, rate limits and slow periods. If your app calls one model with no plan B, their incident is your incident. A fallback chain tries the primary model, and on specific errors (timeouts, 5xx, 429 after retries, overload) moves to a backup: another region, another model, or another provider.

Fallbacks need prompt compatibility (the backup must handle your prompt and tools), clear timeouts per attempt, and logging so you know how often you fell back. A circuit breaker stops hammering a failing provider for a cool-down period.

Do not fall back on errors that will fail everywhere, like invalid requests or content too long.

Why it matters

Your availability can be higher than any single provider’s only if you design for their failures.

Words to know

  • Circuit breakerTemporarily stops calls to a failing dependency.
  • Retryable errorAn error that might succeed on another attempt.

Try it without code

List your failure modes

  1. Read your provider’s error codes page.
  2. Mark which ones should retry, fall back, or fail immediately.

Use it in code

Fallback chain with per-attempt timeouts.

python
import anthropic, openai

CHAIN = [("anthropic", "claude-sonnet-5-5"), ("anthropic", "claude-haiku-5-5"), ("openai", "gpt-5-mini")]
RETRYABLE = (anthropic.APITimeoutError, anthropic.RateLimitError, anthropic.InternalServerError,
             openai.APITimeoutError, openai.RateLimitError, openai.InternalServerError)

def complete(prompt):
    for provider, model in CHAIN:
        try:
            if provider == "anthropic":
                m = anthropic.Anthropic(timeout=20, max_retries=1).messages.create(
                    model=model, max_tokens=500, messages=[{"role": "user", "content": prompt}])
                return model, m.content[0].text
            r = openai.OpenAI(timeout=20, max_retries=1).chat.completions.create(
                model=model, messages=[{"role": "user", "content": prompt}])
            return model, r.choices[0].message.content
        except RETRYABLE as e:
            print(f"fallback from {model}: {type(e).__name__}")
    raise RuntimeError("all models failed")
PRACTICE · TICK IT AFTER THIS

Simulate a primary outage (wrong API key or tiny timeout) and confirm requests succeed via fallback with a log line.

Common mistakes

  • Falling back on 400 errors.
  • Backup model never tested with your prompts and tools.

Go deeper