The idea
Rate limiting caps how many requests a client can make in a time window: 20 chat messages per minute per user, 1,000 API calls per hour per key. It protects your servers from overload, your LLM budget from abuse, and other users from noisy neighbours.
Common algorithms: fixed window (simple, bursty at edges), sliding window (smoother) and token bucket (allows short bursts up to a capacity). Store counters in Redis so all instances share them. Return 429 with Retry-After and rate limit headers so clients can back off.
Limit on several keys at once: IP for anonymous traffic, user ID, API key, and tenant.
A public AI endpoint without rate limits is an open invitation to drain your API credits.
Words to know
- WindowThe time period a limit applies to.
- Retry-AfterHeader telling the client when to try again.
Try it without code
Set limits
- Decide limits for anonymous, free and paid users.
- Write the 429 message each will see.
Use it in code
Sliding-window rate limit as a FastAPI dependency with Redis.
import time, uuid, redis
from fastapi import Depends, HTTPException
r = redis.Redis()
def rate_limit(user_id: str, limit=20, window=60):
key, now = f"rl:{user_id}", time.time()
pipe = r.pipeline()
pipe.zremrangebyscore(key, 0, now - window)
pipe.zadd(key, {uuid.uuid4().hex: now})
pipe.zcard(key)
pipe.expire(key, window)
count = pipe.execute()[2]
if count > limit:
raise HTTPException(429, "Too many messages. Try again in a minute.",
headers={"Retry-After": str(window)})
# @app.post("/chat")
# def chat(body: ChatIn, user=Depends(current_user)):
# rate_limit(user); ...Add per-user and per-IP limits to your chat endpoint and test them with a quick loop of requests.
Common mistakes
- In-memory limits that reset per instance.
- No Retry-After, so clients hammer you.