Area 05 · AI System Design · Concept 16 of 20

Provider Rate Limits

Requests and tokens per minute are capped. Handle it with queues, backoff and spreading load.

11–20 · Production levelPhase 6 · Weeks 21–23≈ 45 min

The idea

Model APIs cap usage per organisation and model: requests per minute, input tokens per minute and output tokens per minute. Exceed one and you get HTTP 429 errors, often with a retry-after header.

Handle it at several layers: client-side rate limiting so you rarely hit the cap, retries with exponential backoff and jitter that respect retry-after, queues to absorb bursts, and spreading load across models, regions or keys where allowed. Request higher limits as you grow.

Monitor remaining-limit headers and 429 rates, and alert before users notice.

Why it matters

At scale, rate limits rather than model quality become the cause of user-facing errors.

Words to know

  • RPM / TPMRequests or tokens per minute.
  • 429HTTP status for "too many requests".

Try it without code

Read your limits

  1. Open your provider console’s limits page.
  2. Note RPM and TPM for your main model.
  3. Compute the max chats per minute at your average prompt size.

Use it in code

A token-bucket limiter shared by all calls in a process.

python
import asyncio, time

class TokenBucket:
    def __init__(self, per_minute):
        self.capacity = self.tokens = per_minute
        self.rate = per_minute / 60
        self.updated = time.monotonic()
        self.lock = asyncio.Lock()

    async def take(self, n):
        async with self.lock:
            while True:
                now = time.monotonic()
                self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
                self.updated = now
                if self.tokens >= n:
                    self.tokens -= n; return
                await asyncio.sleep((n - self.tokens) / self.rate)

itpm = TokenBucket(per_minute=400_000)
# before each call: await itpm.take(estimated_input_tokens)
PRACTICE · TICK IT AFTER THIS

Run 200 concurrent requests through the limiter and confirm zero 429s with throughput near your limit.

Common mistakes

  • Retrying 429s immediately in a tight loop.
  • Per-process limiters with many processes and no shared state.

Go deeper