Learning outcomes
- Calculate token acceptance probabilities and expected speedup ratios
- Evaluate memory-bandwidth vs compute-bound speculative generation workloads
Mental model
Standard auto-regressive LLM generation is memory-bandwidth bound: generating $K$ tokens requires $K$ separate model forward passes, reloading billions of parameters from VRAM for every single token.
Speculative Decoding breaks this memory bottleneck by pairing a small, fast draft model (e.g., Llama-3-8B-Draft) with a large, high-capacity target model (e.g., Llama-3-70B). The draft model speculatively generates $K$ candidate tokens auto-regressively at high speed. The target model then evaluates all $K$ candidate tokens concurrently in a single forward pass, accepting valid tokens and resampling from the target distribution when a draft token diverges.
Theory
Let $p(x)$ be the probability distribution of the target model and $q(x)$ be the probability distribution of the draft model. For each proposed draft token $x_i$:
Acceptance Probability = Min(1.0, p(x_i) / q(x_i))
If token $x_i$ is accepted, execution proceeds to token $x_$. If rejected at position $j$, candidate tokens beyond $j$ are discarded, a new token is sampled from the adjusted distribution $\max(0, p(x) - q(x))$, and the KV-cache is rolled back to position $j$.
Key Architecture Components:
- Draft Speculator: A lightweight model operating on the same tokenizer vocabulary as the target model.
- Target Verifier: The full-scale model running a parallel batch evaluation over $K$ candidate positions.
- Acceptance Sampler: Ensures the final output distribution matches the target model exactly without mathematical approximation errors.
Speedup Factor = (1 + Expected Accepted Tokens) / (1 + Draft Overhead Ratio)
Alternatives and trade-offs
Speculative decoding provides 1.8x–3.2x throughput speedups on memory-bound workloads (low batch size, long outputs) with zero loss in generation quality. However, when draft model acceptance rate drops below 40% (e.g., highly creative or domain-specific generation), draft overhead can degrade throughput below standard target generation.
Failure modes and misconceptions
- Draft Vocabulary Mismatch: Attempting speculative decoding between models with different tokenizers causes alignment failures.
- Low Acceptance Ratio ($\gamma < 0.4$): Poorly matched draft models waste compute verifying rejected tokens.
- VRAM Overhead: Hosting both draft and target models on GPU VRAM reduces maximum batch size capacity.
Knowledge check
What enables speculative decoding to achieve 2x+ inference speedups without altering the output text distribution?
Decision scenario
An enterprise LLM serving platform experiences high latency on 70B parameter code generation requests with batch size 1. The team evaluates adding an 8B draft model for speculative decoding.
Learning outcomes
- Explain how speculative decoding overcomes memory-bandwidth bottlenecks in auto-regressive inference.
- Calculate token acceptance probabilities and expected speedup ratios.
- Select optimal draft model pairings for specialized inference workloads.
Trade-offs
Speculative decoding increases token generation speed by 2x–3x at small batch sizes, but consumes additional GPU VRAM to host the draft model and offers diminishing returns under high compute-bound batch loads.
Evidence assessment
Theory and decision mastery
Decision scenario
An engineering team serves a 70B LLM for real-time customer support chat (Batch Size = 1 to 4).
Which inference optimization strategy provides the highest latency reduction without altering output accuracy?
Primary sources
- Building Effective AI Agents — Anthropic, verified 2026-07-21
- vLLM AsyncLLM engine at v0.10.2 — vLLM Project, verified 2026-07-21