Area 05 · AI System Design · Concept 08 of 20

Throughput vs Latency

More requests per second (big batches) vs a faster single reply. You trade one for the other.

01–10 · The basePhase 6 · Weeks 21–23≈ 45 min

The idea

A GPU generating for one user is underused: decoding is limited by reading the weights from memory, and reading them once can serve many sequences. Batching many requests together raises total throughput (tokens per second across all users) a lot.

But bigger batches make each step a little slower and requests wait to join, so per-user latency rises. Serving is a dial between them: chat apps want low latency at moderate throughput; offline jobs want maximum throughput and accept slow individual requests.

Measure both with a load test at realistic concurrency, and set targets for each workload separately.

Why it matters

It explains GPU serving costs and why offline batch work is so much cheaper.

Picture it

Words to know

  • ThroughputTotal tokens or requests processed per second.
  • ConcurrencyNumber of requests in flight at once.

Try it without code

Read a benchmark

  1. Find a vLLM or provider benchmark that reports latency at several concurrency levels.
  2. Note where latency starts rising sharply.

Use it in code

A tiny async load test against an OpenAI-compatible endpoint.

python
import asyncio, time, statistics
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="x")

async def one():
    t = time.perf_counter()
    r = await client.chat.completions.create(model="Qwen/Qwen2.5-7B-Instruct", max_tokens=128,
        messages=[{"role": "user", "content": "Write a haiku about GPUs."}])
    return time.perf_counter() - t, r.usage.completion_tokens

async def load(concurrency, total=64):
    sem = asyncio.Semaphore(concurrency); start = time.perf_counter()
    async def run():
        async with sem: return await one()
    res = await asyncio.gather(*(run() for _ in range(total)))
    wall = time.perf_counter() - start
    print(concurrency, f"p50 {statistics.median(r[0] for r in res):.2f}s",
          f"throughput {sum(r[1] for r in res) / wall:.0f} tok/s")

for c in (1, 8, 32):
    asyncio.run(load(c))
PRACTICE · TICK IT AFTER THIS

Run the load test at 1, 8 and 32 concurrency against a local vLLM server and chart latency vs throughput.

Common mistakes

  • Benchmarking with one request and planning capacity from it.
  • Unrealistic prompt and output lengths in tests.

Go deeper