Alerting

Good alerting tells you what matters (SLO violations) not what changed (CPU went up). LLM infrastructure needs alerts on latency SLOs, quality degradation, GPU failures, and cost anomalies — with runbooks that guide on-call engineers to resolution.

---

1. SLO Definition

Start with Service Level Objectives — the promises your system makes:

SLO	Target	Measurement window	Error budget
Availability	99.9%	30-day rolling	43 min downtime/month
TTFT P99	<2.0s	5-min windows	0.1% of requests can exceed
TPOT P99	<50ms	5-min windows	0.1% of requests can exceed
Error rate	<0.1%	1-hour rolling	1 in 1000 requests
Quality (LLM-judge)	>4.0/5.0	Daily sample	N/A (monitor, not alert)

From SLOs to Alerts

Don’t alert when P99 TTFT > 2s once. Alert when you’re consuming error budget too fast (burn-rate alerting):

# Prometheus alerting rules
groups:
  - name: llm-slos
    rules:
      # Fast burn: consuming 14.4× error budget (1h window)
      # → Will exhaust 30-day budget in 2 days
      - alert: TTFTBurnRateCritical
        expr: |
          (
            1 - (
              sum(rate(llm_ttft_seconds_bucket{le="2.0"}[1h]))
              / sum(rate(llm_ttft_seconds_count[1h]))
            )
          ) > (14.4 * 0.001)
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "TTFT SLO burn rate critical"
          description: "P99 TTFT budget burning 14.4× too fast. Will exhaust in 2 days."
          runbook: "https://wiki/runbooks/ttft-burn-rate"

      # Slow burn: consuming 3× error budget (6h window)
      # → Will exhaust 30-day budget in 10 days
      - alert: TTFTBurnRateWarning
        expr: |
          (
            1 - (
              sum(rate(llm_ttft_seconds_bucket{le="2.0"}[6h]))
              / sum(rate(llm_ttft_seconds_count[6h]))
            )
          ) > (3 * 0.001)
        for: 15m
        labels:
          severity: warning

---

2. Alert Categories

Infrastructure Alerts

# GPU hardware failures
- alert: GPUDoublebitECCError
  expr: DCGM_FI_DEV_ECC_DBE_AGGREGATE > 0
  for: 0s  # Immediate — hardware failure
  labels:
    severity: critical
  annotations:
    summary: "GPU {{ $labels.gpu }} has uncorrectable ECC errors"
    runbook: "Drain node, replace GPU. Data corruption possible."

- alert: GPUTemperatureThrottling
  expr: DCGM_FI_DEV_GPU_TEMP > 83
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "GPU {{ $labels.gpu }} at {{ $value }}°C (throttling threshold: 83°C)"

- alert: GPUMemoryNearFull
  expr: DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE) > 0.98
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "GPU memory at {{ $value | humanizePercentage }} — OOM risk"

Service Alerts

# Queue building up (requests waiting too long)
- alert: InferenceQueueBacklog
  expr: llm_queue_depth > 50
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Inference queue depth {{ $value }} — requests waiting"
    runbook: "Scale up replicas or investigate slow requests."

# Error rate spike
- alert: InferenceErrorRate
  expr: |
    rate(llm_requests_total{status="error"}[5m]) 
    / rate(llm_requests_total[5m]) > 0.01
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Error rate {{ $value | humanizePercentage }} (SLO: <0.1%)"

# KV cache thrashing
- alert: KVCachePreemptionHigh
  expr: rate(llm_kv_cache_preemptions_total[5m]) > 10
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "KV cache preemptions: {{ $value }}/s — reduce max concurrent requests"

Cost Alerts

# Daily spend anomaly
- alert: DailySpendAnomaly
  expr: |
    sum(increase(llm_tokens_generated_total[1d])) / 1e6 * 1.5 
    > avg_over_time(sum(increase(llm_tokens_generated_total[1d]))[7d:1d]) / 1e6 * 1.5 * 1.5
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "Daily token spend 50% above 7-day average"

---

3. Quality Degradation Detection

Model quality can degrade without errors — the model still responds, but answers get worse:

async def quality_alert_check():
    """Sample production traffic and check answer quality."""
    # Sample 1% of requests
    samples = await get_recent_requests(sample_rate=0.01, window="1h")
    
    quality_scores = []
    for sample in samples:
        score = await llm_judge.evaluate(
            query=sample.prompt,
            response=sample.completion,
            criteria="helpfulness, accuracy, safety",
        )
        quality_scores.append(score)
    
    avg_quality = mean(quality_scores)
    
    # Compare against baseline
    baseline = await get_baseline_quality(sample.model)
    if avg_quality < baseline * 0.9:  # 10% quality drop
        alert(
            severity="warning",
            summary=f"Quality degradation detected: {avg_quality:.2f} vs baseline {baseline:.2f}",
            details=f"Based on {len(samples)} sampled requests in last hour",
        )

---

4. Runbook Design

Every alert must link to a runbook that guides diagnosis and resolution:

# Runbook: TTFT Burn Rate Critical

## Alert
TTFT P99 SLO error budget burning at 14.4× rate.

## Impact
Users experiencing slow first-token response (>2s for >0.1% of requests).

## Diagnosis Steps
1. **Check queue depth**: `llm_queue_depth{model="XXX"}`
   - High queue → too many requests, need to scale or shed load
2. **Check prefill latency**: `llm_prefill_duration_seconds{quantile="0.99"}`
   - Slow prefill → long prompts, check prefix cache hit rate
3. **Check GPU utilization**: `DCGM_FI_DEV_GPU_UTIL`
   - 100% → compute-bound, need more GPUs
   - <50% → scheduling issue, check batch size
4. **Check for bad requests**: Long prompts, unusual models
   - `topk(10, llm_input_tokens_count)` — find outlier requests

## Remediation
- **If queue backlog**: Scale up replicas (`kubectl scale --replicas=N`)
- **If prefix cache cold**: Wait 5-10min for cache to warm, verify system prompts unchanged
- **If GPU-bound**: Enable INT4 quantization, reduce max_tokens default
- **If single-tenant causing**: Apply per-tenant rate limit

## Escalation
If not resolved in 30 min → page on-call SRE lead

---

5. Alert Routing

# Alertmanager configuration
route:
  receiver: 'default'
  group_by: ['alertname', 'model']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
      repeat_interval: 15m
    
    - match:
        severity: warning
      receiver: 'slack-warnings'
      repeat_interval: 2h

receivers:
  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: '<PD_SERVICE_KEY>'
        severity: critical
  
  - name: 'slack-warnings'
    slack_configs:
      - api_url: '<SLACK_WEBHOOK>'
        channel: '#llm-alerts'
        title: '{{ .GroupLabels.alertname }}'
        text: '{{ .CommonAnnotations.summary }}'

Alert Fatigue Prevention

Practice	Implementation
Group related alerts	group_by: [alertname, model] — one notification per incident
Use burn rates, not thresholds	Alert on SLO trajectory, not momentary spikes
Minimum alert duration	for: 5m — ignore transient blips
Actionable only	Every alert has a runbook with concrete steps
Review weekly	Delete alerts nobody acts on

---

Self-Check

• Define SLOs for TTFT, TPOT, availability, and error rate
• Write burn-rate alerting rules that detect slow SLO erosion
• Design GPU hardware alerts for ECC errors, temperature, and memory
• Implement quality degradation detection using LLM-as-judge sampling
• Write a runbook template with diagnosis steps and remediation actions

---

References

• Google SRE Book: Alerting on SLOs
• Prometheus Alertmanager
• For metric definitions, see Metrics