Area 01 · LLM · Concept 05 of 20

KV Cache

Saved keys and values of earlier tokens, so each new token doesn’t redo the whole prompt.

01–10 · The basePhase 1 · Weeks 3–6≈ 1 hour

The idea

During generation the model produces one token at a time. Without help, it would recompute attention keys and values for the entire prompt at every step. The KV cache stores those keys and values after they are computed once, so each new token only computes its own and reads the rest from memory.

This splits a request into two phases. Prefill processes the whole prompt in parallel and fills the cache; it is fast per token. Decode then generates tokens one by one, each reading the cache; it is slower per token and limited by memory bandwidth.

The cache is large. It grows with sequence length, number of layers and model width, and it lives in GPU memory. On a serving machine, the KV cache, not the weights, often decides how many users can be served at once.

Why it matters

It explains why output tokens cost more than input tokens, why time to first token depends on prompt length, and why prompt caching and PagedAttention exist.

Picture it

Words to know

  • PrefillProcessing the prompt and filling the cache.
  • DecodeGenerating new tokens one at a time using the cache.

Try it without code

Feel prefill vs decode

  1. In a chat app, send a very long document with the question "Reply with OK".
  2. Then send a short prompt asking for a 500-word essay.
  3. The first starts slowly and ends fast; the second starts fast and streams for a while.

Use it in code

Generate with and without the cache in Hugging Face and time both.

python
import time
from transformers import AutoTokenizer, AutoModelForCausalLM

name = "Qwen/Qwen2.5-0.5B"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)
ids = tok("Explain the KV cache. " * 50, return_tensors="pt").input_ids

for use_cache in (True, False):
    t = time.time()
    model.generate(ids, max_new_tokens=64, do_sample=False, use_cache=use_cache)
    print(f"use_cache={use_cache}: {time.time() - t:.1f}s")
PRACTICE · TICK IT AFTER THIS

Estimate KV cache size for an 8B model (32 layers, 8 KV heads, head size 128, 16-bit) at 8,000 tokens: 2 × layers × kv_heads × head_dim × tokens × 2 bytes. Write the answer in your notes.

Common mistakes

  • Sizing GPU memory for the weights only and running out once real users send long prompts.
  • Treating input and output tokens as equally expensive.

Go deeper