The idea
Without limits, one user pasting a book or an agent looping can consume a huge share of your spend. Token budgets set caps at several levels: max input and output per request, tokens per user per day, and spend per feature per month.
Enforce them before the call (count input tokens, reject or truncate), during the call (max_tokens) and after (record usage against the user’s budget). Store counters in Redis with expiring keys for daily windows.
Tell users clearly when they hit a limit and what they can do. Budgets also give you predictable costs per plan tier.
Budgets protect your bill and your fair-use policy, and make pricing tiers possible.
Words to know
- QuotaAn allowance of usage over a time window.
Try it without code
Design tiers
- Write daily token limits for free, pro and team plans.
- Estimate cost per user per month at each limit.
Use it in code
Daily per-user token budget in Redis.
import datetime, redis
r = redis.Redis()
DAILY = {"free": 50_000, "pro": 1_000_000}
def key(user_id):
return f"tok:{user_id}:{datetime.date.today().isoformat()}"
def check(user_id, plan, estimated):
used = int(r.get(key(user_id)) or 0)
if used + estimated > DAILY[plan]:
raise PermissionError(f"Daily limit reached ({used}/{DAILY[plan]} tokens). Resets at midnight UTC.")
def record(user_id, usage):
k = key(user_id)
r.incrby(k, usage.input_tokens + usage.output_tokens)
r.expire(k, 60 * 60 * 48)Add per-user daily budgets to your app and a /usage endpoint that shows remaining tokens.
Common mistakes
- Only limiting output tokens.
- Counting after the call but never checking before.