Area 05 · AI System Design · Concept 12 of 20

Speculative Decoding

A small draft model guesses a few tokens; the big model checks them all in one pass.

11–20 · Production levelPhase 6 · Weeks 21–23≈ 1 hour

The idea

Decoding is slow because each token needs a full pass of the big model. Speculative decoding uses a small, fast draft model (or extra prediction heads, or n-gram lookup) to propose the next few tokens. The big model then verifies all of them in a single forward pass.

Tokens the big model agrees with are accepted; at the first disagreement it substitutes its own token and drafting restarts. With the standard acceptance rule, the output distribution is exactly the big model’s, so quality is unchanged.

Speed-ups depend on how often the draft is right: high for predictable text like code and structured output, lower for creative text.

Why it matters

It cuts latency without changing quality, and many providers and engines use it behind the scenes.

Picture it

Words to know

  • Draft modelThe small model that proposes tokens.
  • Acceptance rateShare of drafted tokens the big model keeps.

Try it without code

Guess the next words

  1. Read a sentence aloud and pause; a friend guesses the next 3 words.
  2. Count how many they get right.
  3. That ratio is the acceptance rate idea.

Use it in code

Enable n-gram speculative decoding in vLLM (no draft model needed).

bash
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 4, "prompt_lookup_max": 4}'
# Then rerun your latency test; code and repetitive outputs benefit most.
PRACTICE · TICK IT AFTER THIS

Benchmark TTFT and tokens/s with and without speculative decoding for code generation and for creative writing.

Common mistakes

  • Expecting the same gain on every workload.
  • A draft model with a different tokenizer from the target.

Go deeper