The idea
Your bill is the sum of model tokens (all calls, including retries, judges and rewrites), embeddings, rerankers, vector database, compute and observability. Cost per request divides that by requests served; cost per feature does it per product surface.
Track it from traces: every model call already reports tokens, so tag spans with feature and user and aggregate. Infra costs can be allocated per request on top.
Watch the distribution as well as the mean: a few agent runs that loop can cost more than thousands of normal requests.
It decides pricing, margins and which optimisations are worth doing.
Words to know
- Unit economicsRevenue and cost per unit of usage.
Try it without code
Estimate a feature
- List every model call one request of your RAG app makes.
- Estimate tokens for each and price them.
- Add up the cost per request and per 10,000 requests.
Use it in code
Aggregate cost by feature from logged calls.
import pandas as pd
calls = pd.DataFrame([
{"request": "r1", "feature": "chat", "usd": 0.0031},
{"request": "r1", "feature": "chat", "usd": 0.0004}, # query rewrite
{"request": "r2", "feature": "summary", "usd": 0.0120},
{"request": "r3", "feature": "chat", "usd": 0.0029},
])
per_request = calls.groupby(["feature", "request"]).usd.sum()
print(per_request.groupby("feature").describe()[["count", "mean", "max"]])Add feature tags to your traces and produce a weekly table of requests, mean cost, p95 cost and total per feature.
Common mistakes
- Forgetting judge and retry calls.
- Only tracking the monthly invoice total.