What Is Speculative Decoding? Faster LLM Token Generation Explained

Speculative decoding is a lossless shortcut through autoregressive generation. Instead of running the full target model once per token, a fast drafter guesses several tokens ahead; the target verifies them in a single forward pass and keeps the longest matching prefix. Throughput rises because weight memory traffic drops when guesses land.

Quick answers

Why decode is the bottleneck

Speculative decoding explained as a fast draft path verified by a slower check step

During prefill, GPUs process many prompt tokens in parallel. During decode, the batch is often one new token per request. Every decode step still touches all layers and the growing KV cache, so memory bandwidth and kernel launch overhead dominate. That is why tokens per second falls off a cliff after the first answer token — not because arithmetic vanished, but because the machine loads the full weight set repeatedly for thin work.

Quantization (guide) shrinks those loads. Batching helps datacenters. Speculative decoding attacks the serial loop itself, as introduced in Leviathan et al.'s paper.

How assisted generation works

Hugging Face documents the pattern under decoding strategies:

1. A drafter (smaller model, or extra heads) proposes k candidate tokens.
2. The target runs one forward pass on the concatenated prefix plus drafts.
3. Verification compares target logits at each draft position. Matching tokens are accepted; the first mismatch is resampled from the target distribution; rejected tail drafts are discarded.
4. KV cache updates for all accepted positions, then the loop continues.

The drafter must share the same tokenizer as the target; otherwise byte-level drafts misalign and acceptance collapses. HF notes assisted generation is limited to certain decoding strategies and may not support batched inputs in all versions — check release notes before baking it into production.

The Hugging Face API accepts an assistant_model argument on generate() alongside the target tensors. You load a second AutoModelForCausalLM checkpoint, move it to the same device, and pass it into generate. The call site stays one line; the complexity lives in picking a drafter with aligned vocabulary and acceptable acceptance on your prompts.

The assistant checkpoint is often an order of magnitude smaller — classic pairing is a 7B target with a 125M–1B drafter trained on similar data.

Acceptance rate is the whole game

Inference speed gauge with a lightning needle marking faster decode throughput

Define acceptance rate as the average fraction of drafted tokens kept per round. Speedup roughly scales with how many serial target steps you avoid, minus drafter cost.

Coding agents can show bimodal acceptance: boilerplate accepts well; rare API symbols reject often. Benchmark speculative setups on your system prompts, not a generic wiki article.

Watermarks and auditing tools must account for speculative paths too. Meta TextSeal2 explicitly tests survival when drafts change token timing — a reminder that observability features are not automatic with every speed hack.

Variants beyond a second checkpoint

Prompt lookup decoding (also in the HF doc) reuses overlapping n-grams between prompt and likely output — strong for summarization where the answer copies phrasing from the source. Pass prompt_lookup_num_tokens instead of an assistant model.

Medusa-style multi-head drafting attaches several decoding heads to the target trunk. EAGLE family methods (paper) learn draft features from hidden states. Engines like vLLM and SGLang integrate vendor-specific speculative kernels; flags and supported model lists change every release, so treat engine docs as authoritative.

Block diffusion drafters (for example DFlash) propose fixed-size blocks verified in parallel — useful on Apple Silicon via DFlash-MLX. The idea is the same: parallelize verification, shrink serial depth.

Contrasted with tensor parallelism alone

Tensor parallel (TP) splits layers across GPUs so each device holds fewer weights per forward pass. That helps when one model does not fit on a card. Speculative decoding instead reduces how many forwards you need for a given answer length. TP and speculation solve different constraints; hyperscalers often combine TP for model fit with speculation for user-visible latency on chatty workloads. On a single consumer GPU, TP is usually irrelevant, while a strong drafter can still help if VRAM fits both checkpoints or uses head-based drafting.

Running it yourself

Hugging Face Transformers

Install matching transformers and accelerate, load target plus assistant_model, call generate as above. Start with greedy decoding until you measure acceptance, then try sampling if your engine allows it. Watch GPU memory — you hold two models or extra heads.

llama.cpp and Ollama

llama.cpp documents speculative execution with a draft GGUF and target GGUF. Parameters include draft model path and draft count. Ollama may expose similar flags depending on version; verify the release notes for your build.

MLX on Mac

MLX guide covers standard generation; pair it with community draft projects when you need Metal-optimized speculative stacks. Expect manual model pairing until mlx-lm absorbs first-party flags.

vLLM / SGLang servers

Production stacks expose speculative config in server launch arguments — draft model name, number of speculative tokens, sometimes custom CUDA graphs. Pair with KV cache tuning because accepting multiple tokens per round jumps cache writes. If you see preemption under load, speculative mode may be disabled first.

How this interacts with continuous batching and agents

Serving frameworks batch independent user requests to keep GPUs full. Speculative decoding was originally described for single-sequence assisted generation; datacenter schedulers may fuse or serialize draft steps differently per request. When you load-test a coding agent behind vLLM, measure p95 time-to-last-token with speculative mode on, not kernel micro-benchmarks alone.

Agent harnesses amplify variance: tool JSON, markdown fences, and copied stack traces create repetitive prefixes (good for prompt caching) but irregular token tails (harder for drafters). If your agent emits long unified diffs, acceptance on the first fifty tokens may be high while tail tokens drag — profile both regions.

Multimodal targets add another constraint: draft models must track the same modality stack or stay text-only while the target handles images separately. Mixed setups are immature; most teams speculative-accelerate text-only targets first.

Limits and failure modes

Not a free lunch on prefill-heavy jobs. RAG prompts that spend seconds on prefill but return two-sentence answers see little decode gain.

VRAM stacking. Two full models do not fit where one barely fit. Use a tiny drafter or head-based methods.

Distribution shift. Fine-tuned targets with generic drafters reject more. Fine-tune or distill the drafter on domain text.

Security and compliance. Faster generation does not reduce indirect prompt injection risk. Agents that draft unsafe tool calls still need AgentBeam-class guardrails at execution time.

Evaluation traps. Benchmarks that only measure prefill (some retrieval suites) mis-rank speculative stacks. Use end-to-end task latency on representative chats.

Choosing whether to enable it

Enable speculative decoding when:

• Answers are long (assistant tokens dominate latency).
• You already run a GPU or Metal server with headroom for a drafter.
• You can pair tokenizers and pin engine versions.

Skip or defer when:

• You are memory-bound on a single 8 GB card.
• Traffic is batch-saturated and the scheduler disables assisted paths.
• You need deterministic regression tests across framework upgrades — add a golden acceptance test first.

For apples-to-apples numbers, compare vLLM vs SGLang with speculative flags on and off on the same hardware, same quant, same context.

Myths that waste engineering time

Myth: speculative decoding replaces quantization. Quantization attacks memory bandwidth per weight byte. Speculation attacks step count. You still want Q4 or fp8 weights on edge devices; add speculation only when a drafter fits.

Myth: any small model can draft for any large model. Tokenizer parity is necessary but not sufficient. Domain shift between draft training and target behavior still crushes acceptance.

Myth: speedups transfer 1:1 from papers to your chat UI. UI batching, streaming parsers, and client-side filters sit after the server. Measure end-to-end.

Myth: speculative decode breaks safety filters. Filters run on final tokens; verify policies still see the same stream ordering your compliance team expects.

Build checklist

1. Pin target and drafter versions; record tokenizer fingerprint.
2. Log acceptance rate per route (chat vs codegen vs summarization).
3. Cap max draft length so VRAM spikes stay bounded alongside KV cache limits.
4. Add a regression test that compares assisted vs baseline outputs on 50 golden prompts.
5. Document a kill switch in production configs when acceptance drops after a model swap.

Assisted generation APIs, llama.cpp speculative flags, and engine integrations are accurate as of November 8, 2026. Verify against your installed versions before shipping.

Related on explainx.ai

• KV cache memory explained
• Tokens per second (TPS)
• DFlash-MLX on Apple Silicon
• vLLM vs SGLang
• What is llama.cpp?
• Meta TextSeal2 and speculative paths