Common Pitfalls

Most published benchmarks are misleading — not because of fraud, but because of methodology mistakes. Here are the most common pitfalls and how to avoid them.

---

1. The Pitfalls

Pitfall	Why it’s wrong	How to fix
Testing at zero load	No queuing, no batching competition — unrealistically fast	Benchmark at realistic request rates
Fixed input/output lengths	Hides prefill/decode interference, KV cache pressure	Use ShareGPT or production distribution
No warmup	First requests hit JIT, graph capture, cold cache	30-60s warmup before measuring
Reporting only averages	Hides P99 spikes (which users actually feel)	Report P50, P90, P95, P99
Closed-loop only	Hides queuing latency (new request waits for old)	Use open-loop at target rate
Different hardware	Comparing A100 results to H100 results	Normalize per GPU or match hardware
Different quantization	INT4 vs BF16 “throughput comparison”	State quantization clearly
Ignoring TTFT	Only reporting TPOT or throughput	Report both TTFT and TPOT
Single-run results	Variance from GC, thermal, other processes	Run 3+ times, report variance
Micro-benchmark extrapolation	Kernel-level results don’t predict end-to-end	Always measure end-to-end serving

---

2. The “Fastest Engine” Trap

“Engine X is 2× faster than Engine Y” usually means:

• Tested on a different model size
• Different quantization
• Different batch size / concurrency
• Different hardware
• Different metric (throughput vs latency)

Fair comparison requires: Same model, same quantization, same hardware, same input distribution, same request rate, same metrics.

---

3. The Performance Regression Check

For CI/CD:

# Performance gate: block merge if P95 TPOT regresses > 5%
performance_gate:
  metric: tpot_p95
  baseline: previous_release
  threshold: 1.05  # 5% regression allowed
  test:
    model: meta-llama/Llama-3.1-8B
    dataset: sharegpt
    request_rate: 10
    duration: 120s

For measurement methodology, see Benchmarking Methodology.

---

Self-Check

• Identify at least 5 common benchmarking pitfalls
• Design a fair comparison between two inference engines
• Set up a performance regression gate for CI/CD
• Explain why “Engine X is 2× faster” claims need scrutiny