Area 05 · AI System Design · Concept 04 of 20

Prefix Caching

Reuse the computation for a shared prompt start (same system prompt, same document).

01–10 · The basePhase 6 · Weeks 21–23≈ 45 min

The idea

When many requests start with identical tokens (same system prompt, same tools, same document), the model’s prefill work for that prefix is identical too. Prefix caching stores the KV cache for the prefix and reuses it, skipping most of the prefill.

With hosted APIs this is prompt caching (you mark breakpoints or it happens automatically). With self-hosted engines like vLLM and SGLang, automatic prefix caching shares KV blocks between requests on the server.

Design prompts for it: stable content first, variable content last, no timestamps or request IDs at the top.

Why it matters

Lower time to first token and lower cost for chat over documents, agents and anything with a long shared prompt.

Words to know

  • PrefixThe identical leading tokens shared by several requests.

Try it without code

Reorder a prompt

  1. Take a prompt template with a date at the top.
  2. Move all dynamic parts to the end.
  3. Count how many tokens are now shared across requests.

Use it in code

Turn on automatic prefix caching in vLLM.

bash
pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct --enable-prefix-caching --max-model-len 16384
# Requests sharing the same system prompt + document now reuse cached KV blocks.
# Compare time-to-first-token for the 1st and 2nd request with the same long prefix.
PRACTICE · TICK IT AFTER THIS

Send 20 questions over the same long document to a server with and without prefix caching; compare TTFT.

Common mistakes

  • Dynamic text at the start of prompts.
  • Expecting cache hits across different model versions.

Go deeper