Metrics Definition

Inference performance requires precise metrics. Using the wrong metric — or measuring the right metric incorrectly — leads to optimizing the wrong thing.

---

1. Latency Metrics

Metric	Definition	Formula	Matters for
TTFT	Time from request submission to first output token	$T_{\text{queue}} + T_{\text{prefill}}$	User-perceived responsiveness
TPOT	Average time between output tokens (decode phase)	$(T_{\text{end}} - T_{\text{first}}) / (N_{\text{out}} - 1)$	Streaming generation speed
ITL	Inter-token latency (per-token, not averaged)	$T_{i+1} - T_i$ for each consecutive pair	Detecting jitter/spikes
E2E	Total request latency	$\text{TTFT} + \text{TPOT} \times (N_{\text{out}} - 1)$	Overall experience

TTFT vs TPOT

• Short outputs (20 tokens): TTFT dominates → optimize prefill
• Long outputs (500+ tokens): TPOT dominates → optimize decode
• Streaming chat: Both matter — user sees TTFT first, then TPOT determines “typing speed”

---

2. Throughput Metrics

Metric	Definition	Unit	Perspective
Output tokens/s	Total output tokens generated per second (all requests)	tok/s	System capacity
Tokens/s/GPU	Throughput normalized by GPU count	tok/s/GPU	Hardware efficiency
Requests/s (RPS)	Completed requests per second	req/s	API capacity

Throughput-Latency Tradeoff

As concurrent requests increase:

1. Throughput rises (batching utilizes idle compute)
2. Latency stays flat (batch fits in resources)
3. Latency rises (queuing starts, resources saturated)
4. Throughput plateaus or drops (preemption, OOM)

The operating point: Highest throughput where P99 latency meets SLA.

---

3. Goodput

$$\text{Goodput} = \text{RPS} \times \Pr(\text{latency meets SLA})$$

Raw throughput that doesn’t meet latency targets is useless. Goodput is the metric that maps to user experience and business value.

---

4. Metric Relationships

If you see…	It means…	Optimize…
High TTFT, low TPOT	Prefill is slow (long prompts or queuing)	Prefill throughput, chunked prefill
Low TTFT, high TPOT	Decode is slow (memory-bound)	Quantization, batching, TP
High P99 but low P50	Tail latency from interference	PD disaggregation
Throughput plateaus early	Memory exhaustion (KV cache)	KV quantization, PagedAttention

For detailed latency analysis, see Latency Metrics. For roofline analysis, see Roofline Analysis.

---

Self-Check

• Define TTFT, TPOT, ITL, E2E with formulas
• Determine which metric dominates for a given output length
• Calculate goodput from RPS and SLA compliance
• Diagnose a bottleneck from metric patterns