Metrics

You can’t optimize what you don’t measure. LLM infrastructure metrics span three layers: GPU hardware (utilization, memory, temperature), inference service (TTFT, TPOT, throughput, queue depth), and business (cost per token, requests per model, error rate). Getting the right metrics at the right granularity is the foundation of everything else in observability.

---

1. GPU Hardware Metrics (DCGM)

NVIDIA’s Data Center GPU Manager (DCGM) exposes hardware metrics via the DCGM Exporter for Prometheus:

# dcgm-exporter configuration
# Metrics collected every 10s, exported on :9400/metrics
DCGM_FI_DEV_GPU_UTIL           # GPU compute utilization (0-100%)
DCGM_FI_DEV_MEM_COPY_UTIL      # Memory bandwidth utilization (0-100%)
DCGM_FI_DEV_FB_USED            # Framebuffer (HBM) memory used (MB)
DCGM_FI_DEV_FB_FREE            # Framebuffer memory free (MB)
DCGM_FI_DEV_GPU_TEMP           # GPU temperature (°C)
DCGM_FI_DEV_POWER_USAGE        # Power consumption (W)
DCGM_FI_DEV_PCIE_TX_THROUGHPUT # PCIe TX bandwidth (KB/s)
DCGM_FI_DEV_PCIE_RX_THROUGHPUT # PCIe RX bandwidth (KB/s)
DCGM_FI_DEV_NVLINK_BANDWIDTH_TX # NVLink TX bandwidth (KB/s)
DCGM_FI_DEV_ECC_SBE_AGGREGATE  # Single-bit ECC errors (cumulative)
DCGM_FI_DEV_ECC_DBE_AGGREGATE  # Double-bit ECC errors (critical!)
DCGM_FI_DEV_XID_ERRORS         # XID error codes (GPU faults)

Key GPU Metrics for LLM Serving

Metric	What it tells you	Healthy range	Alert threshold
GPU utilization	SM activity	60-95%	<30% (wasting GPUs) or 100% sustained
Memory utilization	HBM usage	80-95%	>98% (risk of OOM)
Memory BW utilization	HBM bandwidth saturation	40-80%	>90% (bandwidth-bound)
Temperature	Thermal status	40-75°C	>83°C (throttling imminent)
Power	Energy consumption	200-700W (model-dependent)	>TDP (throttling)
ECC DBE count	Hardware reliability	0	>0 (GPU failing, replace)

DCGM Exporter Deployment (Kubernetes)

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: dcgm-exporter
spec:
  selector:
    matchLabels:
      app: dcgm-exporter
  template:
    spec:
      containers:
      - name: dcgm-exporter
        image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.0-3.4.0-ubuntu22.04
        ports:
        - containerPort: 9400
        securityContext:
          privileged: true  # Required for GPU access
        volumeMounts:
        - name: nvidia
          mountPath: /usr/lib/x86_64-linux-gnu
      volumes:
      - name: nvidia
        hostPath:
          path: /usr/lib/x86_64-linux-gnu

---

2. Inference Service Metrics

These are the metrics your SLOs are built on:

from prometheus_client import Histogram, Counter, Gauge

# Latency metrics (histograms for percentile calculation)
TTFT = Histogram(
    "llm_ttft_seconds", "Time to first token",
    ["model", "quantization"],
    buckets=[0.1, 0.25, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0],
)

TPOT = Histogram(
    "llm_tpot_seconds", "Time per output token (inter-token latency)",
    ["model"],
    buckets=[0.01, 0.02, 0.03, 0.05, 0.1, 0.2],
)

E2E_LATENCY = Histogram(
    "llm_e2e_latency_seconds", "End-to-end request latency",
    ["model"],
    buckets=[0.5, 1, 2, 5, 10, 30, 60, 120],
)

# Throughput metrics
TOKENS_GENERATED = Counter(
    "llm_tokens_generated_total", "Total tokens generated",
    ["model", "type"],  # type: prompt, completion
)

REQUESTS = Counter(
    "llm_requests_total", "Total requests",
    ["model", "status"],  # status: success, error, timeout
)

# Queue metrics
QUEUE_DEPTH = Gauge(
    "llm_queue_depth", "Current requests waiting in queue",
    ["model"],
)

BATCH_SIZE = Histogram(
    "llm_batch_size", "Current batch size",
    ["model"],
    buckets=[1, 2, 4, 8, 16, 32, 64, 128, 256],
)

# Resource metrics
KV_CACHE_USAGE = Gauge(
    "llm_kv_cache_usage_pct", "KV cache memory utilization",
    ["model"],
)

PREFIX_CACHE_HIT_RATE = Gauge(
    "llm_prefix_cache_hit_rate", "Prefix cache hit rate",
    ["model"],
)

Instrumenting vLLM/SGLang

# vLLM exposes metrics at /metrics endpoint
# Configure Prometheus to scrape:
scrape_configs:
  - job_name: 'vllm'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['vllm-instance-1:8000', 'vllm-instance-2:8000']
    scrape_interval: 10s

---

3. Business Metrics

Metric	Formula	Purpose
Cost per 1M tokens	GPU cost/hour ÷ tokens/hour × 1M	Pricing, budgeting
Revenue per GPU-hour	Requests × price ÷ GPU-hours	ROI tracking
Requests per model	Count by model label	Capacity planning
Error rate	Errors ÷ total requests	SLO tracking
Tokens per request (avg)	Total tokens ÷ requests	Usage patterns

# PromQL: Cost per million tokens
# Assuming $3/GPU-hour for H100
cost_per_m_tokens = (
    3.0 /  # $/GPU-hour
    rate(llm_tokens_generated_total[1h]) * 1_000_000 /
    3600  # seconds per hour
)

---

4. Collection Pipeline

GPU (DCGM) ──→ DCGM Exporter ──→ Prometheus ──→ Grafana dashboards
                                       ↑              ↓
App (vLLM)  ──→ /metrics endpoint ─────┘        Alertmanager
                                                      ↓
Custom      ──→ OpenTelemetry SDK ──→ OTLP ──→  PagerDuty/Slack

Prometheus Scrape Configuration

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'dcgm'
    static_configs:
      - targets: ['dcgm-exporter:9400']
    scrape_interval: 10s

  - job_name: 'vllm'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: vllm
        action: keep
    scrape_interval: 10s

  - job_name: 'gateway'
    static_configs:
      - targets: ['api-gateway:9090']

---

5. Dashboard Design

Essential Grafana Panels

Row 1: Request Overview
  ├── Requests/sec by model (stacked area)
  ├── Error rate (line, with SLO threshold)
  └── Active requests (gauge)

Row 2: Latency
  ├── TTFT P50/P95/P99 (line chart, per model)
  ├── TPOT P50/P95/P99 (line chart)
  └── E2E latency distribution (heatmap)

Row 3: Throughput
  ├── Tokens/sec generated (stacked area by model)
  ├── Batch size distribution (histogram)
  └── Queue depth (line, with alert threshold)

Row 4: GPU Health
  ├── GPU utilization per instance (multi-line)
  ├── GPU memory usage per instance (multi-line with max)
  ├── GPU temperature (multi-line with threshold)
  └── KV cache utilization (gauge per instance)

Row 5: Cost
  ├── Cost per 1M tokens (line, by model)
  ├── Cumulative daily cost (counter)
  └── GPU-hours consumed (stacked area by team/model)

---

Self-Check

• Deploy DCGM Exporter and list the 5 most important GPU metrics for LLM serving
• Instrument an LLM serving endpoint with TTFT, TPOT, and throughput histograms
• Write PromQL queries for P99 TTFT and cost per million tokens
• Design a Grafana dashboard with request, latency, throughput, and GPU panels
• Explain why GPU utilization alone is misleading for inference workloads

---

References

• NVIDIA DCGM Documentation
• Prometheus Client Libraries
• vLLM Metrics Documentation
• For latency metric definitions, see Latency Metrics