Area 08 · Backend · Concept 14 of 20

Rate Limiting

Cap requests per user or key to protect your app and your LLM bill.

11–20 · Production levelPhase 0 + 5 · Weeks 1–2, 18–20≈ 45 min

The idea

Rate limiting caps how many requests a client can make in a time window: 20 chat messages per minute per user, 1,000 API calls per hour per key. It protects your servers from overload, your LLM budget from abuse, and other users from noisy neighbours.

Common algorithms: fixed window (simple, bursty at edges), sliding window (smoother) and token bucket (allows short bursts up to a capacity). Store counters in Redis so all instances share them. Return 429 with Retry-After and rate limit headers so clients can back off.

Limit on several keys at once: IP for anonymous traffic, user ID, API key, and tenant.

Why it matters

A public AI endpoint without rate limits is an open invitation to drain your API credits.

Words to know

  • WindowThe time period a limit applies to.
  • Retry-AfterHeader telling the client when to try again.

Try it without code

Set limits

  1. Decide limits for anonymous, free and paid users.
  2. Write the 429 message each will see.

Use it in code

Sliding-window rate limit as a FastAPI dependency with Redis.

python
import time, uuid, redis
from fastapi import Depends, HTTPException

r = redis.Redis()

def rate_limit(user_id: str, limit=20, window=60):
    key, now = f"rl:{user_id}", time.time()
    pipe = r.pipeline()
    pipe.zremrangebyscore(key, 0, now - window)
    pipe.zadd(key, {uuid.uuid4().hex: now})
    pipe.zcard(key)
    pipe.expire(key, window)
    count = pipe.execute()[2]
    if count > limit:
        raise HTTPException(429, "Too many messages. Try again in a minute.",
                            headers={"Retry-After": str(window)})

# @app.post("/chat")
# def chat(body: ChatIn, user=Depends(current_user)):
#     rate_limit(user); ...
PRACTICE · TICK IT AFTER THIS

Add per-user and per-IP limits to your chat endpoint and test them with a quick loop of requests.

Common mistakes

  • In-memory limits that reset per instance.
  • No Retry-After, so clients hammer you.

Go deeper