Area 04 · Evals · Concept 19 of 20

Cost per Request

Total token + infra spend divided by requests. Track it per feature.

11–20 · Production levelPhase 4 · Weeks 15–17≈ 45 min

The idea

Your bill is the sum of model tokens (all calls, including retries, judges and rewrites), embeddings, rerankers, vector database, compute and observability. Cost per request divides that by requests served; cost per feature does it per product surface.

Track it from traces: every model call already reports tokens, so tag spans with feature and user and aggregate. Infra costs can be allocated per request on top.

Watch the distribution as well as the mean: a few agent runs that loop can cost more than thousands of normal requests.

Why it matters

It decides pricing, margins and which optimisations are worth doing.

Words to know

  • Unit economicsRevenue and cost per unit of usage.

Try it without code

Estimate a feature

  1. List every model call one request of your RAG app makes.
  2. Estimate tokens for each and price them.
  3. Add up the cost per request and per 10,000 requests.

Use it in code

Aggregate cost by feature from logged calls.

python
import pandas as pd

calls = pd.DataFrame([
    {"request": "r1", "feature": "chat", "usd": 0.0031},
    {"request": "r1", "feature": "chat", "usd": 0.0004},   # query rewrite
    {"request": "r2", "feature": "summary", "usd": 0.0120},
    {"request": "r3", "feature": "chat", "usd": 0.0029},
])
per_request = calls.groupby(["feature", "request"]).usd.sum()
print(per_request.groupby("feature").describe()[["count", "mean", "max"]])
PRACTICE · TICK IT AFTER THIS

Add feature tags to your traces and produce a weekly table of requests, mean cost, p95 cost and total per feature.

Common mistakes

  • Forgetting judge and retry calls.
  • Only tracking the monthly invoice total.

Go deeper