CI/CD for Models

Model CI/CD automates the path from checkpoint to deployment. Unlike software CI/CD, models are multi-GB artifacts, evaluation takes GPU-hours, and format conversion (HuggingFace → TensorRT) is its own build step. But the principle is the same: every change passes automated quality gates before reaching production.

---

1. The Model CI/CD Pipeline

New checkpoint saved (or fine-tune completed)
  │
  ├─→ [Trigger] Webhook / filesystem watch / manual
  │
  ├─→ [Stage 1: Eval] Run benchmark suite on GPU cluster
  │     ├── MMLU, HumanEval, GSM8K (standard)
  │     ├── Custom domain benchmarks
  │     └── Safety evaluation (red-team prompts)
  │
  ├─→ [Gate 1: Quality] All metrics above threshold?
  │     ├── Yes → continue
  │     └── No → notify team, block pipeline
  │
  ├─→ [Stage 2: Regression] Compare against production model
  │     ├── Score delta within tolerance?
  │     └── Latency regression within tolerance?
  │
  ├─→ [Gate 2: Regression] No significant regressions?
  │     ├── Yes → continue
  │     └── No → notify team, block pipeline
  │
  ├─→ [Stage 3: Convert] Build serving formats
  │     ├── SafeTensors → AWQ INT4 quantization
  │     ├── SafeTensors → TensorRT-LLM engine
  │     └── SafeTensors → GGUF (if edge deployment)
  │
  ├─→ [Stage 4: Register] Add to model registry as "staging"
  │
  ├─→ [Stage 5: Canary] Deploy to 5% traffic
  │     └── Monitor 24h: latency, errors, quality
  │
  └─→ [Stage 6: Promote] Move to production (100% traffic)

---

2. Pipeline Implementation

GitHub Actions Example

name: Model CI/CD
on:
  workflow_dispatch:
    inputs:
      checkpoint_path:
        description: 'S3 path to model checkpoint'
        required: true
      model_name:
        description: 'Model name for registry'
        required: true

jobs:
  evaluate:
    runs-on: [self-hosted, gpu-h100]
    timeout-minutes: 120
    steps:
      - uses: actions/checkout@v4
      
      - name: Run evaluation suite
        run: |
          lm_eval --model vllm \
            --model_args pretrained=${{ inputs.checkpoint_path }},tensor_parallel_size=4 \
            --tasks mmlu,humaneval,gsm8k,ifeval \
            --batch_size auto \
            --output_path results/eval/
      
      - name: Run custom domain benchmarks
        run: python scripts/eval_custom.py --model ${{ inputs.checkpoint_path }}
      
      - name: Run safety evaluation
        run: python scripts/eval_safety.py --model ${{ inputs.checkpoint_path }}
      
      - name: Check quality gates
        run: |
          python scripts/check_gates.py \
            --results results/eval/ \
            --thresholds configs/quality_gates.yaml \
            --comparison-model production

  convert:
    needs: evaluate
    runs-on: [self-hosted, gpu-h100]
    steps:
      - name: Quantize to AWQ INT4
        run: |
          python scripts/quantize.py \
            --model ${{ inputs.checkpoint_path }} \
            --method awq \
            --bits 4 \
            --output s3://models/${{ inputs.model_name }}/staging/awq-int4/
      
      - name: Build TensorRT-LLM engine
        run: |
          trtllm-build \
            --checkpoint_dir ${{ inputs.checkpoint_path }} \
            --output_dir s3://models/${{ inputs.model_name }}/staging/trt/ \
            --gemm_plugin float16 \
            --max_batch_size 256

  register:
    needs: convert
    runs-on: ubuntu-latest
    steps:
      - name: Register in model registry
        run: |
          python scripts/register_model.py \
            --name ${{ inputs.model_name }} \
            --stage staging \
            --eval-results results/eval/ \
            --artifacts s3://models/${{ inputs.model_name }}/staging/

  canary:
    needs: register
    runs-on: ubuntu-latest
    steps:
      - name: Deploy canary (5% traffic)
        run: |
          kubectl apply -f deployments/canary.yaml
          python scripts/monitor_canary.py --duration 24h --threshold configs/canary_gates.yaml
      
      - name: Promote to production
        if: success()
        run: python scripts/promote.py --name ${{ inputs.model_name }} --stage production

---

3. Regression Tests

Test	What it catches	Threshold	Measurement
MMLU drop	General knowledge regression	<1% drop vs production	Absolute score comparison
HumanEval drop	Code quality regression	<2% drop	Pass@1 comparison
GSM8K drop	Math reasoning regression	<2% drop	Accuracy comparison
Latency regression	Slower inference	P95 TPOT <1.05× production	Benchmark on same hardware
Safety regression	More harmful outputs	Refusal rate within 5% of baseline	Red-team prompt set
Format compliance	JSON/structured output breaks	Valid output rate <2% drop	Format validation tests

# Quality gate checker
def check_gates(new_results: dict, production_results: dict, thresholds: dict) -> list:
    """Check all quality gates, return list of failures."""
    failures = []
    
    for metric, threshold in thresholds.items():
        new_score = new_results.get(metric)
        prod_score = production_results.get(metric)
        
        if threshold["type"] == "absolute":
            if new_score < threshold["min"]:
                failures.append(f"{metric}: {new_score:.3f} < {threshold['min']}")
        
        elif threshold["type"] == "relative":
            max_drop = threshold["max_regression"]
            if prod_score and (prod_score - new_score) > max_drop:
                failures.append(
                    f"{metric}: dropped {prod_score - new_score:.3f} "
                    f"(max allowed: {max_drop})"
                )
    
    return failures

---

4. Format Conversion Pipeline

Each serving engine needs its own model format:

CONVERSION_MATRIX = {
    "vllm": {
        "input": "safetensors",
        "quantize": "awq_int4",       # or gptq, fp8
        "time": "30-60 min (70B)",
        "gpu_required": True,
    },
    "tensorrt_llm": {
        "input": "safetensors",
        "build": "trtllm-build",
        "time": "60-120 min (70B)",
        "gpu_required": True,         # Engine is GPU-specific
    },
    "llama_cpp": {
        "input": "safetensors",
        "convert": "convert_hf_to_gguf.py",
        "quantize": "q4_k_m",
        "time": "15-30 min (70B)",
        "gpu_required": False,        # CPU conversion
    },
}

Caching: Format conversion is deterministic — cache converted artifacts by input hash. If the same checkpoint is re-evaluated, skip conversion.

---

5. Canary Monitoring

class CanaryMonitor:
    """Monitor canary deployment for 24 hours."""
    
    CANARY_GATES = {
        "error_rate": {"max": 0.001, "window": "1h"},
        "p99_ttft_ms": {"max_regression": 1.10, "window": "1h"},  # 10% slower OK
        "p99_tpot_ms": {"max_regression": 1.10, "window": "1h"},
        "quality_score": {"min": 3.5, "window": "4h"},  # LLM-as-judge sample
    }
    
    async def monitor(self, duration_hours: int = 24):
        start = time.time()
        while (time.time() - start) / 3600 < duration_hours:
            metrics = await self.collect_canary_metrics()
            
            for gate, threshold in self.CANARY_GATES.items():
                if not self.check(metrics, gate, threshold):
                    await self.rollback_canary()
                    raise CanaryFailed(f"Canary gate {gate} failed: {metrics[gate]}")
            
            await asyncio.sleep(300)  # Check every 5 minutes
        
        return CanaryResult(passed=True, duration=duration_hours)

---

6. Rollback

Every deployment must be instantly reversible:

async def rollback(model_name: str, reason: str):
    """Instant rollback to previous production model."""
    previous = await registry.get_previous_production(model_name)
    
    # Swap routing (model is already loaded on GPUs from before)
    await router.set_production_model(model_name, previous.version)
    
    # Update registry
    current = await registry.get_production(model_name)
    current.status = "failed"
    previous.status = "production"
    await registry.update(current)
    await registry.update(previous)
    
    # Alert team
    await alert(f"Rollback: {model_name} reverted to {previous.version}. Reason: {reason}")

Pre-condition: Keep the previous production model loaded (or quickly loadable) for at least 48 hours after a new deployment.

For the evaluation suite that powers CI/CD gates, see Offline Evaluation. For model registration, see Model Registry.

---

Self-Check

• Design a 6-stage model CI/CD pipeline with quality gates
• Implement regression tests comparing new model against production baseline
• Set up automated format conversion (SafeTensors → AWQ → TensorRT)
• Build a canary monitor with automatic rollback on gate failure
• Ensure instant rollback capability for any production deployment

---

References

• lm-evaluation-harness
• TensorRT-LLM Build Guide
• For offline evaluation, see Offline Evaluation