The idea
Transient failures (timeouts, 429s, 503s) often succeed on a second try. But retrying immediately, from thousands of clients at once, makes an overloaded service worse. Exponential backoff waits longer after each failure (1s, 2s, 4s, 8s), and jitter adds randomness so clients don’t retry in sync.
Retry only retryable errors, cap the number of attempts and the total time, respect Retry-After headers, and make sure the operation is idempotent. Official LLM SDKs already retry some errors; know their defaults so you don’t stack retries on retries.
Log each retry. A rising retry rate is an early warning of upstream trouble.
It turns brief provider hiccups into invisible blips instead of user-facing errors.
Words to know
- Exponential backoffDoubling the wait after each failure.
- JitterRandom variation added to waits.
Try it without code
Compute waits
- Write the waits for 5 attempts with base 1s and doubling.
- Add ±50% random jitter to each.
- Sum the worst-case total time.
Use it in code
Retries with tenacity: exponential backoff, jitter and only retryable errors.
# pip install tenacity anthropic
import anthropic
from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_random_exponential
client = anthropic.Anthropic(max_retries=0) # we control retries here
RETRYABLE = (anthropic.RateLimitError, anthropic.APITimeoutError,
anthropic.InternalServerError, anthropic.APIConnectionError)
@retry(retry=retry_if_exception_type(RETRYABLE),
wait=wait_random_exponential(multiplier=1, max=30),
stop=stop_after_attempt(5), reraise=True)
def complete(messages):
return client.messages.create(model="claude-sonnet-5-5", max_tokens=500, messages=messages)Wrap all model calls in your app with this retry policy and log attempt counts.
Common mistakes
- Retrying 400 and 401 errors.
- SDK retries plus your retries plus queue retries multiplying.