The idea
When many requests start with identical tokens (same system prompt, same tools, same document), the model’s prefill work for that prefix is identical too. Prefix caching stores the KV cache for the prefix and reuses it, skipping most of the prefill.
With hosted APIs this is prompt caching (you mark breakpoints or it happens automatically). With self-hosted engines like vLLM and SGLang, automatic prefix caching shares KV blocks between requests on the server.
Design prompts for it: stable content first, variable content last, no timestamps or request IDs at the top.
Lower time to first token and lower cost for chat over documents, agents and anything with a long shared prompt.
Words to know
- PrefixThe identical leading tokens shared by several requests.
Try it without code
Reorder a prompt
- Take a prompt template with a date at the top.
- Move all dynamic parts to the end.
- Count how many tokens are now shared across requests.
Use it in code
Turn on automatic prefix caching in vLLM.
pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct --enable-prefix-caching --max-model-len 16384
# Requests sharing the same system prompt + document now reuse cached KV blocks.
# Compare time-to-first-token for the 1st and 2nd request with the same long prefix.Send 20 questions over the same long document to a server with and without prefix caching; compare TTFT.
Common mistakes
- Dynamic text at the start of prompts.
- Expecting cache hits across different model versions.