Acceptance Rate & Speedup Bounds

Speculative decoding isn’t always faster. The actual speedup depends on the acceptance rate, draft cost, verification overhead, and workload characteristics. This chapter provides the formulas to calculate whether speculation helps before deploying.

1. Expected Tokens per Round

With speculation length $\gamma$ and per-position acceptance probability $a$ :

\mathbb{E}[L] = \frac{1 - a^{\gamma+1}}{1 - a}

Acceptance rate $a$	$\gamma = 3$	$\gamma = 5$	$\gamma = 8$
0.5	1.9 tokens	2.0 tokens	2.0 tokens
0.7	2.5	2.9	3.2
0.8	3.0	3.7	4.3
0.9	3.4	4.7	6.1
0.95	3.7	5.3	7.4

Key observation: Diminishing returns from longer speculation. At $a = 0.8$ : going from $\gamma = 3$ to $\gamma = 8$ only adds 1.2 tokens, but costs 5 more draft steps and verification positions.

2. The Speedup Formula

Let $T_0$ = baseline decode time per token. Let $T_{\text{round}}$ = time for one speculative round (draft + verify + overhead):

\text{Speedup} = \frac{T_0 \times \mathbb{E}[L]}{T_{\text{round}}}

Breaking down $T_{\text{round}}$ :

T_{\text{round}} = \underbrace{T_{\text{draft}}}_{\gamma \text{ draft steps}} + \underbrace{T_{\text{verify}}}_{\text{target forward on } \gamma \text{ tokens}} + \underbrace{T_{\text{overhead}}}_{\text{sampling, KV rollback}}

Worked Example: 70B with 8B Draft

Component	Time	Notes
$T_0$ (baseline decode)	5.2 ms	70B, TP=8, single request
$T_{\text{draft}}$ ( $\gamma = 5$ )	2.5 ms	8B draft, 5 autoregressive steps
$T_{\text{verify}}$	6.0 ms	70B forward on 5+1=6 tokens (like a mini-prefill)
$T_{\text{overhead}}$	0.5 ms	Sampling, KV rollback
$T_{\text{round}}$	9.0 ms	Total round time

With $a = 0.8$ , $\gamma = 5$ : $\mathbb{E}[L] = 3.7$ tokens per round.

\text{Speedup} = \frac{5.2 \times 3.7}{9.0} = 2.1\times

2.1× speedup — typical for a well-matched draft. Not the theoretical 4× because the draft and verification aren’t free.

3. Why Verification Isn’t Free

“The target checks $\gamma$ tokens in one forward pass” doesn’t mean verification takes the same time as one decode step:

Factor	Effect on $T_{\text{verify}}$
More tokens → more attention compute	$T_{\text{verify}}$ grows with $\gamma$ (not 1:1, but not free)
Candidate KV cache entries	Must be written then potentially rolled back
Longer context	Attention over full context for each candidate position
Large batch	Less spare compute to absorb extra verification work

Rule of thumb: $T_{\text{verify}} \approx 1.0-1.5 \times T_0$ for $\gamma \leq 5$ on a single request. At $\gamma = 10$ or with large batches, verification overhead grows.

4. When Speculation Doesn’t Help

High Batch Size

At high concurrency, the GPU is already well-utilized by batching. Adding speculative candidates competes for compute rather than utilizing idle resources:

Batch=1:   GPU is 3% utilized → plenty of room for speculation ✅
Batch=32:  GPU is 30% utilized → some room for speculation ⚠️
Batch=128: GPU is 90% utilized → speculation adds overhead ❌

As batch size increases, baseline throughput rises (batching already helps), and the speedup from speculation shrinks toward 1× or even below 1× (net slowdown).

Long Context

Long sequences increase both draft and verification cost:

Draft model must attend to full context at each step
Verification attention grows with context length
Candidate KV cache memory: $B \times \gamma \times m_{\text{KV/token}}$

Poor Acceptance Rate

If acceptance rate drops below ~0.5, speculation rarely helps:

At $a = 0.5$ , $\gamma = 5$ : expect only 2.0 tokens per round
Round takes ~2× baseline time → effective speedup $\approx 1.0$
Draft compute is wasted on mostly-rejected candidates

Short Output

For outputs of $< 10$ tokens, the overhead of setting up draft model + first round may exceed the savings.

5. The Decision Checklist

Before deploying speculative decoding:

Question	Check	If no
Is decode the bottleneck?	TPOT >> TTFT	Don’t bother — optimize prefill instead
Is batch size low (<32)?	GPU compute not saturated	At high batch, batching already helps more
Is acceptance rate > 0.6?	Test with calibration data	Bad draft → net slowdown
Is draft cheap enough?	$T_{\text{draft}} < 0.5 \times T_0$	Consider n-gram or Medusa instead
Is output long enough?	Average output > 20 tokens	Overhead dominates short outputs

Quick Estimation

Before running benchmarks, estimate:

\text{Estimated speedup} = \frac{\mathbb{E}[L]}{1 + T_{\text{draft}} / T_0 + T_{\text{verify}} / T_0}

If this is < 1.3, speculation probably isn’t worth the complexity.

For practical configuration and benchmarking, see Practical Integration.

Self-Check

Calculate expected tokens per round for $a=0.75$ , $\gamma=5$
Calculate speedup for a given $T_0$ , $T_{\text{draft}}$ , $T_{\text{verify}}$ , and acceptance rate
Explain why high batch size reduces speculation benefit
Explain why verification isn’t free (attention, KV, compute)
Use the decision checklist to determine if speculation helps a given workload
Calculate the break-even acceptance rate for a given draft/verify cost