Area 05 · AI System Design · Concept 05 of 20

Token Budgets

Cap input and output tokens per request, per user or per feature.

01–10 · The basePhase 6 · Weeks 21–23≈ 45 min

The idea

Without limits, one user pasting a book or an agent looping can consume a huge share of your spend. Token budgets set caps at several levels: max input and output per request, tokens per user per day, and spend per feature per month.

Enforce them before the call (count input tokens, reject or truncate), during the call (max_tokens) and after (record usage against the user’s budget). Store counters in Redis with expiring keys for daily windows.

Tell users clearly when they hit a limit and what they can do. Budgets also give you predictable costs per plan tier.

Why it matters

Budgets protect your bill and your fair-use policy, and make pricing tiers possible.

Words to know

  • QuotaAn allowance of usage over a time window.

Try it without code

Design tiers

  1. Write daily token limits for free, pro and team plans.
  2. Estimate cost per user per month at each limit.

Use it in code

Daily per-user token budget in Redis.

python
import datetime, redis
r = redis.Redis()
DAILY = {"free": 50_000, "pro": 1_000_000}

def key(user_id):
    return f"tok:{user_id}:{datetime.date.today().isoformat()}"

def check(user_id, plan, estimated):
    used = int(r.get(key(user_id)) or 0)
    if used + estimated > DAILY[plan]:
        raise PermissionError(f"Daily limit reached ({used}/{DAILY[plan]} tokens). Resets at midnight UTC.")

def record(user_id, usage):
    k = key(user_id)
    r.incrby(k, usage.input_tokens + usage.output_tokens)
    r.expire(k, 60 * 60 * 48)
PRACTICE · TICK IT AFTER THIS

Add per-user daily budgets to your app and a /usage endpoint that shows remaining tokens.

Common mistakes

  • Only limiting output tokens.
  • Counting after the call but never checking before.

Go deeper