Benchmarking Methodology

A benchmark is only as good as its methodology. The difference between a misleading and a useful benchmark comes down to load generation, input distribution, measurement period, and reporting.

1. Open-Loop vs Closed-Loop

Mode	How it works	What it measures	Pitfall
Open-loop	Send requests at fixed rate regardless of responses	True system capacity under load	May overwhelm system (queue growth)
Closed-loop	Send next request only after previous completes	Max throughput at specific concurrency	Hides queuing latency, overly optimistic

Open-loop is more realistic — production traffic doesn’t slow down because your server is busy. Always report which mode you used.

# Open-loop benchmark (vLLM)
python benchmark_serving.py \
    --request-rate 10 \          # fixed 10 req/s regardless of completion
    --num-prompts 500

# Closed-loop benchmark
python benchmark_serving.py \
    --request-rate inf \          # send as fast as possible
    --max-concurrency 32          # but limit to 32 concurrent

2. Input Distribution

Distribution	Source	Realistic?
Fixed 128/128	Synthetic (128 input, 128 output)	❌ Unrealistic — real inputs vary wildly
ShareGPT	Real conversations from ShareGPT.com	✅ Good for chat workloads
LMSYS-Chat-1M	Real API logs	✅ Good for API workloads
Synthetic power-law	Generated with realistic length distribution	⚠️ OK if distribution matches production
Production replay	Recorded from your actual traffic	✅ Best (if available)

Always use a realistic distribution. Fixed-length benchmarks hide prefill/decode interference, KV cache pressure, and scheduling effects.

3. Measurement Protocol

1. WARMUP (30-60 seconds)
   - JIT compilation, CUDA graph capture, cache warming
   - DO NOT include in measurements

2. RAMP-UP (optional, 10-30 seconds)
   - Gradually increase to target request rate
   - Stabilize queues and batches

3. STEADY-STATE MEASUREMENT (120+ seconds)
   - Fixed request rate, stable system
   - Collect ALL metrics during this window

4. COOLDOWN
   - Drain remaining requests
   - Include in measurement only if relevant

4. Reporting

Minimum reporting requirements:

What to report	Why
Hardware (GPU model, count, TP degree)	Reproducibility
Model (name, size, quantization)	Comparability
Load (request rate, concurrency, distribution)	Context
P50, P90, P95, P99 for TTFT and TPOT	Tails matter more than averages
Throughput (output tok/s)	System capacity
Goodput at target SLA	Business relevance

Never report only averages. P99 TPOT can be 5× the average — the average hides real user experience.

For metric definitions, see Metrics Definition. For common mistakes, see Common Pitfalls.

Self-Check

Compare open-loop vs closed-loop benchmarking
Choose an input distribution for a chat vs API workload
Design a measurement protocol with proper warmup
List minimum reporting requirements for a reproducible benchmark