The idea
For streamed responses, users judge speed mostly by time to first token (TTFT): how long until something appears. After that, tokens per second matters less as long as text keeps flowing faster than people read.
TTFT is driven by queueing, network, prompt length (prefill), and everything you do before the model call: query rewriting, retrieval, reranking, guardrail checks. Each pre-step adds directly to TTFT.
Reduce it with prefix caching, shorter prompts, parallel pre-steps, faster models for pre-steps, and starting the stream early (for example showing "Searching your documents…" status events).
It is the latency metric most closely tied to how fast your product feels.
Words to know
- TTFTTime to first token.
- TPOTTime per output token after the first.
Try it without code
Time your pipeline
- Write down each step before the model starts generating.
- Estimate milliseconds for each.
- Pick two that could run in parallel.
Use it in code
Measure TTFT and tokens per second from a stream.
import time, anthropic
client = anthropic.Anthropic()
start = time.perf_counter(); first = None; chunks = 0
with client.messages.stream(model="claude-sonnet-5-5", max_tokens=400,
messages=[{"role": "user", "content": "Explain TTFT in 150 words."}]) as s:
for text in s.text_stream:
if first is None:
first = time.perf_counter() - start
chunks += 1
usage = s.get_final_message().usage
total = time.perf_counter() - start
print(f"TTFT {first*1000:.0f} ms, total {total:.1f}s, {usage.output_tokens / (total - first):.0f} tok/s")Measure TTFT for your RAG app, then run retrieval and query rewriting in parallel and measure again.
Common mistakes
- Optimising total time while TTFT stays slow.
- Sequential pre-steps that could run in parallel.