The idea
A GPU generating for one user is underused: decoding is limited by reading the weights from memory, and reading them once can serve many sequences. Batching many requests together raises total throughput (tokens per second across all users) a lot.
But bigger batches make each step a little slower and requests wait to join, so per-user latency rises. Serving is a dial between them: chat apps want low latency at moderate throughput; offline jobs want maximum throughput and accept slow individual requests.
Measure both with a load test at realistic concurrency, and set targets for each workload separately.
It explains GPU serving costs and why offline batch work is so much cheaper.
Picture it
Words to know
- ThroughputTotal tokens or requests processed per second.
- ConcurrencyNumber of requests in flight at once.
Try it without code
Read a benchmark
- Find a vLLM or provider benchmark that reports latency at several concurrency levels.
- Note where latency starts rising sharply.
Use it in code
A tiny async load test against an OpenAI-compatible endpoint.
import asyncio, time, statistics
from openai import AsyncOpenAI
client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="x")
async def one():
t = time.perf_counter()
r = await client.chat.completions.create(model="Qwen/Qwen2.5-7B-Instruct", max_tokens=128,
messages=[{"role": "user", "content": "Write a haiku about GPUs."}])
return time.perf_counter() - t, r.usage.completion_tokens
async def load(concurrency, total=64):
sem = asyncio.Semaphore(concurrency); start = time.perf_counter()
async def run():
async with sem: return await one()
res = await asyncio.gather(*(run() for _ in range(total)))
wall = time.perf_counter() - start
print(concurrency, f"p50 {statistics.median(r[0] for r in res):.2f}s",
f"throughput {sum(r[1] for r in res) / wall:.0f} tok/s")
for c in (1, 8, 32):
asyncio.run(load(c))Run the load test at 1, 8 and 32 concurrency against a local vLLM server and chart latency vs throughput.
Common mistakes
- Benchmarking with one request and planning capacity from it.
- Unrealistic prompt and output lengths in tests.