Area 04 · Evals · Concept 18 of 20

P95 Latency

95% of requests are faster than this. It shows the slow tail that an average hides.

11–20 · Production levelPhase 4 · Weeks 15–17≈ 30 min

The idea

Average latency hides pain. If 90 requests take 1 second and 10 take 20 seconds, the average is 2.9 seconds, but one user in ten waits 20 seconds. Percentiles describe the distribution: p50 is the median, p95 is the time 95% of requests beat, p99 the slowest 1%.

LLM latency has heavy tails: long prompts, long outputs, retries, rate limiting and agent loops all push the tail out. Track p50 and p95 for total time and time to first token, per feature.

Set targets (SLOs) on p95, and alert when it rises.

Why it matters

Users remember the slow requests. p95 is the number that matches their experience.

Words to know

  • PercentileValue below which a given share of observations fall.
  • SLOService level objective, e.g. "p95 under 4 seconds".

Try it without code

Compute by hand

  1. Write 20 response times, including two very slow ones.
  2. Sort them; the 19th value is roughly p95.
  3. Compare with the average.

Use it in code

Percentiles from logged latencies.

python
import numpy as np

latencies_ms = np.array([820, 900, 950, 1010, 1100, 1200, 1250, 1300, 1400, 1500,
                         1600, 1700, 1800, 2000, 2100, 2300, 2600, 3100, 9800, 21000])
print("mean", int(latencies_ms.mean()))
for p in (50, 95, 99):
    print(f"p{p}", int(np.percentile(latencies_ms, p)))
PRACTICE · TICK IT AFTER THIS

Log latency for 100 requests to your app and report p50, p95 and the main cause of the slowest 5.

Common mistakes

  • Reporting only averages.
  • Mixing streaming and non-streaming requests in one number.

Go deeper