Area 01 · LLM · Concept 12 of 20

Prompt Caching

Reuse an unchanged prompt prefix across calls to cut cost and latency.

11–20 · Production levelPhase 1 · Weeks 3–6≈ 45 min

The idea

Many calls share the same start: a long system prompt, tool definitions, a reference document. Prompt caching lets the provider keep the processed state (the KV cache) for that prefix for a few minutes. Later calls that begin with exactly the same tokens read from the cache instead of recomputing.

Cached input tokens are billed at a fraction of the normal price, and time to first token drops, sometimes sharply for long prefixes. Writing to the cache may cost slightly more than a normal input token the first time.

Caching only matches an identical prefix. Put stable content first (system prompt, tools, documents) and variable content last (the user’s question). A timestamp at the top of the prompt breaks every cache hit.

Why it matters

For chat over a long document or agents with many tools, caching is often the single biggest cost saving available.

Words to know

  • Cache hitThe prefix was found and reused.
  • Cache breakpointWhere you mark the end of the cacheable prefix.
  • TTLHow long the cache entry lives.

Try it without code

Spot the cache in usage numbers

  1. Run the code below twice within a minute.
  2. Compare cache_creation_input_tokens on the first run with cache_read_input_tokens on the second.

Use it in code

Mark a long document as cacheable and inspect usage.

python
import anthropic
client = anthropic.Anthropic()
handbook = open("handbook.txt").read()   # a few thousand tokens or more

def ask(q):
    msg = client.messages.create(
        model="claude-sonnet-5-5", max_tokens=300,
        system=[{"type": "text", "text": "Answer using the handbook only."},
                {"type": "text", "text": handbook, "cache_control": {"type": "ephemeral"}}],
        messages=[{"role": "user", "content": q}],
    )
    u = msg.usage
    print(q, "| write:", u.cache_creation_input_tokens, "read:", u.cache_read_input_tokens)

ask("How many leave days do I get?")
ask("What is the expense limit?")
PRACTICE · TICK IT AFTER THIS

Measure cost and latency for 10 questions over one document with and without caching. Record the saving as a percentage.

Common mistakes

  • Putting dynamic values (dates, user names) before the cached content.
  • Prefixes shorter than the provider’s minimum cacheable length.

Go deeper