Speculative Decoding & Verification

Mental model

Standard auto-regressive LLM generation is memory-bandwidth bound: generating $K$ tokens requires $K$ separate model forward passes, reloading billions of parameters from VRAM for every single token.

Speculative Decoding breaks this memory bottleneck by pairing a small, fast draft model (e.g., Llama-3-8B-Draft) with a large, high-capacity target model (e.g., Llama-3-70B). The draft model speculatively generates $K$ candidate tokens auto-regressively at high speed. The target model then evaluates all $K$ candidate tokens concurrently in a single forward pass, accepting valid tokens and resampling from the target distribution when a draft token diverges.

Theory

Let $p(x)$ be the probability distribution of the target model and $q(x)$ be the probability distribution of the draft model. For each proposed draft token $x_i$:

If token $x_i$ is accepted, execution proceeds to token $x_$. If rejected at position $j$, candidate tokens beyond $j$ are discarded, a new token is sampled from the adjusted distribution $\max(0, p(x) - q(x))$, and the KV-cache is rolled back to position $j$.

Key Architecture Components:

• Draft Speculator: A lightweight model operating on the same tokenizer vocabulary as the target model.
• Target Verifier: The full-scale model running a parallel batch evaluation over $K$ candidate positions.
• Acceptance Sampler: Ensures the final output distribution matches the target model exactly without mathematical approximation errors.

Alternatives and trade-offs

Speculative decoding provides 1.8x–3.2x throughput speedups on memory-bound workloads (low batch size, long outputs) with zero loss in generation quality. However, when draft model acceptance rate drops below 40% (e.g., highly creative or domain-specific generation), draft overhead can degrade throughput below standard target generation.

Failure modes and misconceptions

• Draft Vocabulary Mismatch: Attempting speculative decoding between models with different tokenizers causes alignment failures.
• Low Acceptance Ratio ($\gamma < 0.4$): Poorly matched draft models waste compute verifying rejected tokens.
• VRAM Overhead: Hosting both draft and target models on GPU VRAM reduces maximum batch size capacity.

Knowledge check

Decision scenario

An enterprise LLM serving platform experiences high latency on 70B parameter code generation requests with batch size 1. The team evaluates adding an 8B draft model for speculative decoding.

Learning outcomes

• Explain how speculative decoding overcomes memory-bandwidth bottlenecks in auto-regressive inference.
• Calculate token acceptance probabilities and expected speedup ratios.
• Select optimal draft model pairings for specialized inference workloads.

Trade-offs

Speculative decoding increases token generation speed by 2x–3x at small batch sizes, but consumes additional GPU VRAM to host the draft model and offers diminishing returns under high compute-bound batch loads.