Area 08 · Backend · Concept 16 of 20

Retries with Backoff

Retry failed calls with growing waits (plus a little randomness) instead of hammering.

11–20 · Production levelPhase 0 + 5 · Weeks 1–2, 18–20≈ 45 min

The idea

Transient failures (timeouts, 429s, 503s) often succeed on a second try. But retrying immediately, from thousands of clients at once, makes an overloaded service worse. Exponential backoff waits longer after each failure (1s, 2s, 4s, 8s), and jitter adds randomness so clients don’t retry in sync.

Retry only retryable errors, cap the number of attempts and the total time, respect Retry-After headers, and make sure the operation is idempotent. Official LLM SDKs already retry some errors; know their defaults so you don’t stack retries on retries.

Log each retry. A rising retry rate is an early warning of upstream trouble.

Why it matters

It turns brief provider hiccups into invisible blips instead of user-facing errors.

Words to know

  • Exponential backoffDoubling the wait after each failure.
  • JitterRandom variation added to waits.

Try it without code

Compute waits

  1. Write the waits for 5 attempts with base 1s and doubling.
  2. Add ±50% random jitter to each.
  3. Sum the worst-case total time.

Use it in code

Retries with tenacity: exponential backoff, jitter and only retryable errors.

python
# pip install tenacity anthropic
import anthropic
from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_random_exponential

client = anthropic.Anthropic(max_retries=0)      # we control retries here

RETRYABLE = (anthropic.RateLimitError, anthropic.APITimeoutError,
             anthropic.InternalServerError, anthropic.APIConnectionError)

@retry(retry=retry_if_exception_type(RETRYABLE),
       wait=wait_random_exponential(multiplier=1, max=30),
       stop=stop_after_attempt(5), reraise=True)
def complete(messages):
    return client.messages.create(model="claude-sonnet-5-5", max_tokens=500, messages=messages)
PRACTICE · TICK IT AFTER THIS

Wrap all model calls in your app with this retry policy and log attempt counts.

Common mistakes

  • Retrying 400 and 401 errors.
  • SDK retries plus your retries plus queue retries multiplying.

Go deeper