The idea
APIs charge per token with no idle cost. GPUs charge per hour whether busy or not. Self-hosting wins when you have high, steady volume that keeps GPUs busy, a model small enough to run efficiently, or requirements (privacy, residency, custom fine-tunes) that APIs cannot meet.
To compare, compute your effective cost per million tokens on GPUs: hourly price ÷ (measured tokens per second × 3,600 × utilisation) × 1,000,000. Utilisation is the killer: a GPU busy 20% of the time costs five times more per token than one busy 100%.
Add engineering and operations time, redundancy (at least two GPUs for availability), and the quality gap between the open model and the API model.
It is one of the biggest financial decisions in an AI product and a frequent interview question.
Words to know
- UtilisationShare of time the GPU does useful work.
- Break-even volumeUsage at which both options cost the same.
Try it without code
Build the spreadsheet
- Column A: daily tokens. Column B: API cost. Column C: GPU cost at measured throughput.
- Find the break-even row.
Use it in code
Break-even calculator.
def gpu_cost_per_mtok(gpu_usd_per_hour, tokens_per_sec, utilisation):
return gpu_usd_per_hour / (tokens_per_sec * 3600 * utilisation) * 1_000_000
api = 1.00 # USD per million tokens for a comparable API model (blended)
for util in (0.1, 0.3, 0.6, 0.9):
gpu = gpu_cost_per_mtok(gpu_usd_per_hour=2.0, tokens_per_sec=1500, utilisation=util)
print(f"util {util:.0%}: GPU {gpu:.2f} USD/MTok vs API {api:.2f} -> {'GPU' if gpu < api else 'API'}")Using your vLLM benchmark numbers, write a one-page recommendation for 10M, 100M and 1B tokens per day.
Common mistakes
- Comparing at 100% utilisation.
- Ignoring staff time and redundancy.