Draft Strategies — infra.bea.ai

The draft strategy determines speculative decoding’s effectiveness. A good draft is fast (cheap to generate), accurate (high acceptance rate), and compatible (same tokenizer, same context). This chapter covers the full spectrum from separate models to self-draft approaches.

1. What Makes a Good Draft

Property	Why it matters	Tradeoff
Speed	Draft time adds to every round	Faster draft = more net speedup
Accuracy	Higher acceptance rate = more tokens per round	Better draft often means larger/slower model
Compatibility	Must share tokenizer and produce valid continuations	Limits which models can pair

The acceptance rate $a$ is the probability each candidate is accepted. With acceptance rate $a$ and $\gamma$ candidates:

\mathbb{E}[\text{tokens per round}] = \frac{1 - a^{\gamma+1}}{1 - a}

At $a = 0.8$ , $\gamma = 5$ : expected 3.7 tokens per round. At $a = 0.5$ : only 2.0 tokens.

2. Separate Draft Model

The simplest approach: use a smaller model from the same family.

# vLLM with separate draft model
vllm serve meta-llama/Llama-3.1-70B \
    --speculative-model meta-llama/Llama-3.1-8B \
    --num-speculative-tokens 5 \
    --speculative-draft-tensor-parallel-size 1

Consideration	Details
Speed	~5-10× faster than target (smaller model)
Accuracy	60-85% acceptance rate (same family helps)
Compatibility	Must share tokenizer; same family strongly preferred
Resources	Separate GPU memory for draft model weights + KV cache

Not any small model works: A 1B model drafting for 70B may have poor acceptance rate because the distribution mismatch is too large. Typically, draft should be 5-10× smaller (8B for 70B, 1B for 8B).

3. N-gram and Prompt Lookup

No draft model at all — find candidates from the existing context:

N-gram Matching

Search the already-generated text for n-grams that match the current suffix:

Generated so far: "The attention mechanism allows the model to focus on"
Current suffix (last 3 tokens): "to focus on"

Search history for "to focus on" → found at position 42:
  "to focus on relevant parts of the input"

Candidate: "relevant parts of the input" (next 5 tokens after match)

Pros: Zero compute cost, zero extra memory. Cons: Only works when the model repeats itself (code, repetitive text). Acceptance rate varies wildly — 90%+ for repetitive code, <10% for creative text.

# vLLM with n-gram speculation
vllm serve meta-llama/Llama-3.1-70B \
    --speculative-model [ngram] \
    --num-speculative-tokens 5 \
    --ngram-prompt-lookup-max 4

Prompt Lookup

Similar but searches the input prompt rather than generated text. Useful when the output contains phrases from the input (summarization, extraction).

4. Medusa: Multi-Head Self-Draft

Medusa adds extra prediction heads to the target model itself — no separate model needed:

Standard LM:     hidden_state → LM_head → logits → 1 next token

Medusa:          hidden_state → LM_head    → logits → token t+1 (standard)
                              → Medusa_head_1 → logits → token t+2 (speculative)
                              → Medusa_head_2 → logits → token t+3 (speculative)
                              → Medusa_head_3 → logits → token t+4 (speculative)

Each Medusa head predicts a future token position from the current hidden state, without autoregressive dependency between heads.

Tree Attention

Since each head independently predicts multiple candidates, the candidates form a tree rather than a single chain:

Position t+1: [A, B]          (2 candidates from head 1)
Position t+2: [C, D, E]       (3 candidates from head 2)
Position t+3: [F, G]          (2 candidates from head 3)

Tree structure (12 paths):
  A → C → F
  A → C → G
  A → D → F
  ...
  B → E → G

Verify all 12 paths in one forward pass using tree attention mask

Tree attention lets the target verify all paths simultaneously — the attention mask ensures each path only sees its ancestors, not siblings.

Trade-offs

Aspect	Medusa
Extra parameters	Small (one linear layer per head, ~0.5% of model)
Training required	Yes — heads must be trained on target model’s representations
Accuracy	Moderate (50-70%) — heads predict independently, miss inter-token dependencies
No separate model	✅ No extra memory for draft model

5. EAGLE: Feature-Level Prediction

EAGLE improves on Medusa by predicting in feature space rather than token space:

Medusa:  hidden_state_t → predict token_{t+1}, token_{t+2}, ... independently

EAGLE:   hidden_state_t + token_t → small autoregressive network →
         predict hidden_state_{t+1} → predict hidden_state_{t+2} → ...
         Then: each predicted hidden_state → LM_head → candidate token

Key difference: EAGLE’s draft network is autoregressive in feature space — each predicted hidden state depends on the previous one. This captures inter-token dependencies that Medusa’s independent heads miss.

EAGLE Versions

Version	Innovation	Improvement
EAGLE	Autoregressive feature prediction	Higher acceptance than Medusa
EAGLE-2	Confidence-based tree budget allocation	Spend more verification on likely paths
EAGLE-3	Improved training + richer input features	Further accuracy gains

Important: EAGLE checkpoints are model-specific. An EAGLE head trained for Llama-3.1-70B won’t work with Llama-3.1-8B or a different 70B model. Always verify checkpoint compatibility.

6. Choosing a Draft Strategy

Draft Strategies

Strategy	Speed	Accuracy	Extra memory	Training needed	Best for
Separate model	Fast (small model)	Good (same family)	Draft model weights	No	General purpose
N-gram	Free	Variable (0-90%)	None	No	Repetitive text, code
Prompt lookup	Free	Variable	None	No	Summarization, extraction
Medusa	Very fast (heads only)	Moderate	~0.5% extra params	Yes	Low-latency, no extra memory
EAGLE	Fast	High	Small network	Yes	Best quality, single-user latency

For how to measure acceptance rate and determine if speculative decoding helps your workload, see Acceptance Rate & Speedup Bounds.

Self-Check

Calculate expected tokens per round for a given acceptance rate and speculation length
Explain why draft model should be from the same family as target
Describe n-gram matching and when it works well vs poorly
Compare Medusa’s independent heads vs EAGLE’s autoregressive feature prediction
Explain tree attention and why siblings can’t see each other
Choose a draft strategy for: (a) general chat, (b) code completion, (c) latency-critical API