The idea
Model APIs have outages, rate limits and slow periods. If your app calls one model with no plan B, their incident is your incident. A fallback chain tries the primary model, and on specific errors (timeouts, 5xx, 429 after retries, overload) moves to a backup: another region, another model, or another provider.
Fallbacks need prompt compatibility (the backup must handle your prompt and tools), clear timeouts per attempt, and logging so you know how often you fell back. A circuit breaker stops hammering a failing provider for a cool-down period.
Do not fall back on errors that will fail everywhere, like invalid requests or content too long.
Your availability can be higher than any single provider’s only if you design for their failures.
Words to know
- Circuit breakerTemporarily stops calls to a failing dependency.
- Retryable errorAn error that might succeed on another attempt.
Try it without code
List your failure modes
- Read your provider’s error codes page.
- Mark which ones should retry, fall back, or fail immediately.
Use it in code
Fallback chain with per-attempt timeouts.
import anthropic, openai
CHAIN = [("anthropic", "claude-sonnet-5-5"), ("anthropic", "claude-haiku-5-5"), ("openai", "gpt-5-mini")]
RETRYABLE = (anthropic.APITimeoutError, anthropic.RateLimitError, anthropic.InternalServerError,
openai.APITimeoutError, openai.RateLimitError, openai.InternalServerError)
def complete(prompt):
for provider, model in CHAIN:
try:
if provider == "anthropic":
m = anthropic.Anthropic(timeout=20, max_retries=1).messages.create(
model=model, max_tokens=500, messages=[{"role": "user", "content": prompt}])
return model, m.content[0].text
r = openai.OpenAI(timeout=20, max_retries=1).chat.completions.create(
model=model, messages=[{"role": "user", "content": prompt}])
return model, r.choices[0].message.content
except RETRYABLE as e:
print(f"fallback from {model}: {type(e).__name__}")
raise RuntimeError("all models failed")Simulate a primary outage (wrong API key or tiny timeout) and confirm requests succeed via fallback with a log line.
Common mistakes
- Falling back on 400 errors.
- Backup model never tested with your prompts and tools.