The idea
No single trick fixes cost; you stack them. Measure first: cost per feature and per request from traces. Then apply in rough order of effort: trim prompts and outputs (ask for concise answers, cap max_tokens), prompt caching, semantic caching, routing to smaller models, batch APIs for offline work, and finally distillation or self-hosting for high-volume narrow tasks.
Re-run evals after every change. A saving that drops quality is not a saving.
Keep a cost dashboard per feature with targets, and review it as often as you review latency and errors.
AI features often launch with poor margins. Systematic optimisation is how they become sustainable.
Picture it
Words to know
- Cost per successful taskTotal cost divided by tasks that actually met the quality bar.
Try it without code
Find the top three
- List your features by monthly cost.
- For the top one, pick two optimisations from the list and estimate savings.
Use it in code
Estimate stacked savings before building anything.
base = 12_000 # USD per month for the feature today
steps = [("trim prompts 20%", 0.20), ("prompt caching on 60% of input", 0.25),
("route 50% to small model (5x cheaper)", 0.40), ("semantic cache 15% hits", 0.15)]
cost = base
for name, saving in steps:
cost *= 1 - saving
print(f"{name:40s} -> {cost:8.0f} USD/month")Apply two optimisations to your RAG app, re-run evals, and report cost per request and quality before and after.
Common mistakes
- Optimising features that are a tiny share of spend.
- Skipping evals after a cost change.