The idea
Model APIs cap usage per organisation and model: requests per minute, input tokens per minute and output tokens per minute. Exceed one and you get HTTP 429 errors, often with a retry-after header.
Handle it at several layers: client-side rate limiting so you rarely hit the cap, retries with exponential backoff and jitter that respect retry-after, queues to absorb bursts, and spreading load across models, regions or keys where allowed. Request higher limits as you grow.
Monitor remaining-limit headers and 429 rates, and alert before users notice.
At scale, rate limits rather than model quality become the cause of user-facing errors.
Words to know
- RPM / TPMRequests or tokens per minute.
- 429HTTP status for "too many requests".
Try it without code
Read your limits
- Open your provider console’s limits page.
- Note RPM and TPM for your main model.
- Compute the max chats per minute at your average prompt size.
Use it in code
A token-bucket limiter shared by all calls in a process.
import asyncio, time
class TokenBucket:
def __init__(self, per_minute):
self.capacity = self.tokens = per_minute
self.rate = per_minute / 60
self.updated = time.monotonic()
self.lock = asyncio.Lock()
async def take(self, n):
async with self.lock:
while True:
now = time.monotonic()
self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
self.updated = now
if self.tokens >= n:
self.tokens -= n; return
await asyncio.sleep((n - self.tokens) / self.rate)
itpm = TokenBucket(per_minute=400_000)
# before each call: await itpm.take(estimated_input_tokens)Run 200 concurrent requests through the limiter and confirm zero 429s with throughput near your limit.
Common mistakes
- Retrying 429s immediately in a tight loop.
- Per-process limiters with many processes and no shared state.