Benchmarking Tools

Use the right tool for the right measurement. This chapter covers the practical tools for inference benchmarking — from quick latency tests to full production load profiling.

---

1. vLLM Built-in Benchmarks

# Offline throughput benchmark
vllm bench throughput \
    --model meta-llama/Llama-3.1-70B \
    --input-len 512 --output-len 128 \
    --num-prompts 1000 \
    --tensor-parallel-size 4

# Online serving benchmark
vllm bench serve \
    --model meta-llama/Llama-3.1-70B \
    --dataset sharegpt \
    --request-rate 10 \
    --num-prompts 500

# Single-request latency
vllm bench latency \
    --model meta-llama/Llama-3.1-70B \
    --input-len 128 --output-len 128 \
    --batch-size 1

---

2. GenAI-Perf (NVIDIA)

Full-featured inference benchmarking with automatic metric collection:

genai-perf \
    --model llama-3.1-70b \
    --endpoint localhost:8000/v1/completions \
    --request-rate 10 \
    --warmup 60 \
    --measurement-interval 300 \
    --streaming

# Output includes:
# TTFT: P50, P90, P95, P99
# TPOT: P50, P90, P95, P99
# Output tokens/s
# Input tokens/s
# Request latency distribution

---

3. Custom Benchmark Scripts

import asyncio, time, aiohttp

async def benchmark(url, prompts, rate):
    results = []
    async with aiohttp.ClientSession() as session:
        for prompt in prompts:
            start = time.perf_counter()
            first_token_time = None
            
            async with session.post(url, json={
                "model": "llama-3.1-70b",
                "prompt": prompt,
                "max_tokens": 200,
                "stream": True,
            }) as resp:
                async for chunk in resp.content:
                    if first_token_time is None:
                        first_token_time = time.perf_counter()
            
            end = time.perf_counter()
            results.append({
                "ttft": first_token_time - start,
                "e2e": end - start,
            })
            await asyncio.sleep(1 / rate)  # open-loop rate control
    
    return results

---

4. Profiling Tools (for Debugging)

Tool	What it measures	When to use
torch.profiler	Operator-level GPU time	Identify slow kernels
Nsight Systems	Full timeline (CPU + GPU + NCCL)	Scheduling gaps, comm overlap
Nsight Compute	Single kernel analysis	Optimize specific kernel
py-spy	Python CPU profiling	Find Python bottlenecks

For detailed profiling, see PyTorch Profiler and Nsight Systems.

---

Self-Check

• Run vLLM bench with ShareGPT dataset at different request rates
• Use GenAI-Perf to collect TTFT/TPOT percentiles
• Write a custom open-loop benchmark with streaming
• Choose the right profiling tool for a given performance question