The idea
Decoding is slow because each token needs a full pass of the big model. Speculative decoding uses a small, fast draft model (or extra prediction heads, or n-gram lookup) to propose the next few tokens. The big model then verifies all of them in a single forward pass.
Tokens the big model agrees with are accepted; at the first disagreement it substitutes its own token and drafting restarts. With the standard acceptance rule, the output distribution is exactly the big model’s, so quality is unchanged.
Speed-ups depend on how often the draft is right: high for predictable text like code and structured output, lower for creative text.
It cuts latency without changing quality, and many providers and engines use it behind the scenes.
Picture it
Words to know
- Draft modelThe small model that proposes tokens.
- Acceptance rateShare of drafted tokens the big model keeps.
Try it without code
Guess the next words
- Read a sentence aloud and pause; a friend guesses the next 3 words.
- Count how many they get right.
- That ratio is the acceptance rate idea.
Use it in code
Enable n-gram speculative decoding in vLLM (no draft model needed).
vllm serve Qwen/Qwen2.5-7B-Instruct \
--speculative-config '{"method": "ngram", "num_speculative_tokens": 4, "prompt_lookup_max": 4}'
# Then rerun your latency test; code and repetitive outputs benefit most.Benchmark TTFT and tokens/s with and without speculative decoding for code generation and for creative writing.
Common mistakes
- Expecting the same gain on every workload.
- A draft model with a different tokenizer from the target.