Practical Integration

This chapter provides the practical steps: configure speculative decoding in vLLM/SGLang, measure acceptance rate, benchmark against baseline, and tune for your specific workload.

---

1. vLLM Configuration

Separate Draft Model

vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --speculative-model meta-llama/Llama-3.1-8B-Instruct \
    --num-speculative-tokens 5 \
    --speculative-draft-tensor-parallel-size 1

N-gram Speculation

vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --speculative-model [ngram] \
    --num-speculative-tokens 5 \
    --ngram-prompt-lookup-max 4 \
    --ngram-prompt-lookup-min 2

EAGLE

# Requires matching EAGLE checkpoint
vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --speculative-model /path/to/eagle-llama-70b \
    --speculative-model-quantization fp8 \
    --num-speculative-tokens 5

Key Parameters

Parameter	Default	Tuning guidance
--num-speculative-tokens	5	Start at 3-5; increase only if acceptance rate > 0.8
--speculative-draft-tensor-parallel-size	1	Keep at 1 unless draft model is large
--spec-decoding-acceptance-method	rejection_sampler	Use typical_acceptance_sampler for slightly faster but approximate

---

2. Measuring Acceptance Rate

Per-Request Metrics

# vLLM returns acceptance rate in metrics
import requests

response = requests.post("http://localhost:8000/v1/completions", json={
    "model": "meta-llama/Llama-3.1-70B-Instruct",
    "prompt": "Write a Python function to sort a list",
    "max_tokens": 200,
    "temperature": 0.0,
})

# Check vLLM metrics endpoint
metrics = requests.get("http://localhost:8000/metrics").text
# Look for: vllm:spec_decode_draft_acceptance_rate

Prometheus Metrics

# Key speculative decoding metrics in vLLM:
vllm:spec_decode_draft_acceptance_rate          # average acceptance rate
vllm:spec_decode_num_accepted_tokens_total      # total accepted
vllm:spec_decode_num_draft_tokens_total         # total drafted
vllm:spec_decode_num_emitted_tokens_total       # total emitted (accepted + bonus)

Target: acceptance rate > 0.7 for speculation to be worthwhile.

---

3. Benchmarking: Baseline vs Speculative

Step 1: Establish Baseline

# Baseline (no speculation)
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4

# Benchmark
python benchmark_serving.py \
    --model meta-llama/Llama-3.1-70B-Instruct \
    --dataset-name sharegpt \
    --request-rate 1 \
    --num-prompts 200
# Record: TTFT, TPOT, throughput

Step 2: Run with Speculation

# Same model, add speculation
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --speculative-model meta-llama/Llama-3.1-8B-Instruct \
    --num-speculative-tokens 5

# Same benchmark
python benchmark_serving.py \
    --model meta-llama/Llama-3.1-70B-Instruct \
    --dataset-name sharegpt \
    --request-rate 1 \
    --num-prompts 200
# Compare: TTFT, TPOT, throughput

Step 3: Compare at Different Load Levels

# Test at increasing request rates
for rate in 1 2 4 8 16 32; do
    python benchmark_serving.py \
        --request-rate $rate \
        --num-prompts 200
done
# Plot: throughput vs TPOT at each rate, baseline vs speculative

---

4. What to Look For

Metric	Baseline	With speculation	What it means
TPOT (low load)	40 ms	18 ms	✅ 2.2× speedup — speculation working
TPOT (high load)	15 ms	17 ms	❌ No benefit — batching already efficient
TTFT	500 ms	600 ms	⚠️ Slightly worse — draft adds to first step
Throughput (low load)	25 tok/s	55 tok/s	✅ 2.2× throughput gain
Throughput (high load)	800 tok/s	780 tok/s	❌ Slight regression — overhead at saturation
Acceptance rate	—	0.78	Good (> 0.7 threshold)

Decision

• Low-load latency-sensitive: Speculation helps significantly (2-3× TPOT reduction)
• High-load throughput-focused: Speculation doesn’t help (batching already efficient)
• Mixed workload: Evaluate at your actual request rate distribution

---

5. Tuning Guide

If you see…	Try…
Acceptance rate < 0.6	Better draft model (closer to target)
Acceptance rate > 0.9	Increase num-speculative-tokens
TTFT regression	Reduce num-speculative-tokens or use n-gram (no extra TTFT)
Throughput worse at high load	Disable speculation at high concurrency (adaptive)
Memory pressure from draft	Use EAGLE/Medusa (no separate model memory) or smaller draft
Code tasks: high acceptance	N-gram works great, no draft model needed
Creative tasks: low acceptance	Consider disabling speculation

Quality Verification

Speculative decoding with proper rejection sampling is mathematically lossless. But verify:

# Generate same prompts with and without speculation
# Compare: exact match for greedy (temperature=0)
# Compare: distribution similarity for sampling (temperature>0)

For the theoretical foundations, see Core Idea: Draft & Verify. For understanding the speedup formula, see Acceptance Rate & Speedup Bounds.

---

Self-Check

• Configure vLLM with a separate draft model, n-gram, and EAGLE
• Measure acceptance rate from vLLM metrics
• Benchmark speculative decoding against baseline at multiple load levels
• Interpret results: when does speculation help vs hurt
• Apply the tuning guide for a given acceptance rate and workload
• Verify quality: compare outputs with and without speculation

---

References

• vLLM Speculative Decoding Guide
• SGLang Speculative Decoding