Area 05 · AI System Design · Concept 07 of 20

Time to First Token

The wait before the first word appears. This is what users feel as speed.

01–10 · The basePhase 6 · Weeks 21–23≈ 30 min

The idea

For streamed responses, users judge speed mostly by time to first token (TTFT): how long until something appears. After that, tokens per second matters less as long as text keeps flowing faster than people read.

TTFT is driven by queueing, network, prompt length (prefill), and everything you do before the model call: query rewriting, retrieval, reranking, guardrail checks. Each pre-step adds directly to TTFT.

Reduce it with prefix caching, shorter prompts, parallel pre-steps, faster models for pre-steps, and starting the stream early (for example showing "Searching your documents…" status events).

Why it matters

It is the latency metric most closely tied to how fast your product feels.

Words to know

  • TTFTTime to first token.
  • TPOTTime per output token after the first.

Try it without code

Time your pipeline

  1. Write down each step before the model starts generating.
  2. Estimate milliseconds for each.
  3. Pick two that could run in parallel.

Use it in code

Measure TTFT and tokens per second from a stream.

python
import time, anthropic
client = anthropic.Anthropic()

start = time.perf_counter(); first = None; chunks = 0
with client.messages.stream(model="claude-sonnet-5-5", max_tokens=400,
                            messages=[{"role": "user", "content": "Explain TTFT in 150 words."}]) as s:
    for text in s.text_stream:
        if first is None:
            first = time.perf_counter() - start
        chunks += 1
    usage = s.get_final_message().usage
total = time.perf_counter() - start
print(f"TTFT {first*1000:.0f} ms, total {total:.1f}s, {usage.output_tokens / (total - first):.0f} tok/s")
PRACTICE · TICK IT AFTER THIS

Measure TTFT for your RAG app, then run retrieval and query rewriting in parallel and measure again.

Common mistakes

  • Optimising total time while TTFT stays slow.
  • Sequential pre-steps that could run in parallel.

Go deeper