Navigation
🏭 Platform & Infrastructure· 10 min read

Online Evaluation

Canary deployments, shadow mode, A/B testing, real-traffic quality monitoring, automatic rollback, and production quality dashboards.

online-evaluationcanaryshadowa-b-testmonitoringrollback

Online evaluation tests models with real production traffic — the final validation before full deployment and the continuous quality monitor afterward. Offline benchmarks catch obvious regressions; online evaluation catches the subtle failures that only appear with real user queries.

1. Deployment Strategies

StrategyHow it worksRisk to usersWhen to use
Shadow modeNew model runs parallel, output discardedZeroFirst deploy of new architecture
Canary5-10% traffic to new modelLowRoutine model updates
A/B test50/50 split with statistical testingMediumComparing fundamentally different models
Blue/greenFull cutover with instant rollbackMedium-highWhen canary is impractical

2. Shadow Mode

Run the new model alongside production without affecting users:

class ShadowDeployment:
    """Run new model in parallel, compare outputs without serving them."""
    
    def __init__(self, production_model, shadow_model, sample_rate=0.10):
        self.production = production_model
        self.shadow = shadow_model
        self.sample_rate = sample_rate
    
    async def serve(self, request: ChatRequest) -> Response:
        # Always serve from production
        prod_response = await self.production.generate(request)
        
        # Sample subset for shadow comparison (non-blocking)
        if random.random() < self.sample_rate:
            asyncio.create_task(self._shadow_compare(request, prod_response))
        
        return prod_response
    
    async def _shadow_compare(self, request, prod_response):
        """Run shadow model and log comparison."""
        try:
            shadow_response = await asyncio.wait_for(
                self.shadow.generate(request),
                timeout=30,
            )
            
            # Compare quality
            comparison = await self.judge.compare(
                request.prompt, prod_response.text, shadow_response.text
            )
            
            await self.log({
                "request_id": request.id,
                "prod_tokens": prod_response.output_tokens,
                "shadow_tokens": shadow_response.output_tokens,
                "prod_latency_ms": prod_response.latency_ms,
                "shadow_latency_ms": shadow_response.latency_ms,
                "judge_preference": comparison.winner,
                "judge_score_prod": comparison.score_a,
                "judge_score_shadow": comparison.score_b,
            })
        except Exception as e:
            logger.error(f"Shadow comparison failed: {e}")

Shadow Mode Analysis

-- After 24 hours of shadow mode
SELECT 
    count(*) AS total_comparisons,
    avg(CASE WHEN judge_preference = 'shadow' THEN 1.0 ELSE 0.0 END) AS shadow_win_rate,
    avg(judge_score_shadow) - avg(judge_score_prod) AS avg_quality_delta,
    avg(shadow_latency_ms) / avg(prod_latency_ms) AS latency_ratio,
    percentile_cont(0.99) WITHIN GROUP (ORDER BY shadow_latency_ms) AS shadow_p99
FROM shadow_comparisons
WHERE timestamp > now() - interval '24 hours';

-- Results:
-- total_comparisons: 5,000
-- shadow_win_rate: 0.55 (shadow is better 55% of time — promising)
-- avg_quality_delta: +0.3 (on 10-point scale)
-- latency_ratio: 1.05 (5% slower — acceptable)
-- shadow_p99: 3,200ms (within SLO of 5s)

3. Canary Deployment

Gradually shift traffic with automatic rollback:

class CanaryController:
    STAGES = [
        {"pct": 5, "duration_hours": 4, "name": "initial"},
        {"pct": 25, "duration_hours": 8, "name": "expanding"},
        {"pct": 50, "duration_hours": 12, "name": "half"},
        {"pct": 100, "duration_hours": 0, "name": "full"},
    ]
    
    ROLLBACK_CONDITIONS = {
        "error_rate": {"max": 0.005, "window": "15m"},
        "p99_ttft_regression": {"max": 1.20, "window": "30m"},  # 20% regression
        "p99_tpot_regression": {"max": 1.15, "window": "30m"},
        "quality_score_drop": {"max": 0.5, "window": "1h"},  # On 10-point scale
    }
    
    async def run_canary(self, new_model_version: str):
        for stage in self.STAGES:
            logger.info(f"Canary stage: {stage['name']} ({stage['pct']}%)")
            await self.set_traffic_split(new_model_version, stage["pct"])
            
            # Monitor for the stage duration
            end_time = time.time() + stage["duration_hours"] * 3600
            while time.time() < end_time:
                metrics = await self.collect_metrics(new_model_version)
                
                # Check rollback conditions
                for condition, threshold in self.ROLLBACK_CONDITIONS.items():
                    if self.exceeds_threshold(metrics, condition, threshold):
                        logger.critical(f"Canary FAILED: {condition} = {metrics[condition]}")
                        await self.rollback(new_model_version)
                        raise CanaryFailed(condition)
                
                await asyncio.sleep(60)  # Check every minute
            
            logger.info(f"Stage {stage['name']} passed")
        
        logger.info("Canary completed — model promoted to production")

Kubernetes Canary Implementation

# Istio traffic splitting for LLM canary
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: llm-serving
spec:
  hosts:
  - llm.internal
  http:
  - route:
    - destination:
        host: vllm-production
        port: {number: 8000}
      weight: 95
    - destination:
        host: vllm-canary
        port: {number: 8000}
      weight: 5

4. A/B Testing

For significant model changes, run a proper statistical test:

class ABTest:
    def __init__(self, model_a, model_b, min_samples=1000, significance=0.05):
        self.model_a = model_a
        self.model_b = model_b
        self.min_samples = min_samples
        self.significance = significance
    
    async def analyze(self) -> ABResult:
        samples = await self.collect_samples()
        
        if len(samples["a"]) < self.min_samples:
            return ABResult(status="insufficient_data")
        
        from scipy import stats
        
        # Compare quality scores
        quality_a = [s["quality"] for s in samples["a"]]
        quality_b = [s["quality"] for s in samples["b"]]
        
        t_stat, p_value = stats.ttest_ind(quality_a, quality_b)
        
        # Compare user satisfaction
        sat_a = mean(s["thumbs_up"] for s in samples["a"])
        sat_b = mean(s["thumbs_up"] for s in samples["b"])
        
        winner = None
        if p_value < self.significance:
            winner = "b" if mean(quality_b) > mean(quality_a) else "a"
        
        return ABResult(
            quality_a=mean(quality_a),
            quality_b=mean(quality_b),
            p_value=p_value,
            significant=p_value < self.significance,
            winner=winner,
            satisfaction_a=sat_a,
            satisfaction_b=sat_b,
            samples={"a": len(samples["a"]), "b": len(samples["b"])},
        )

Minimum Sample Size

def min_sample_size(effect_size=0.1, power=0.8, significance=0.05):
    """Minimum samples per variant for detecting a given effect size."""
    from scipy.stats import norm
    
    z_alpha = norm.ppf(1 - significance / 2)  # 1.96 for 0.05
    z_beta = norm.ppf(power)                   # 0.84 for 0.80
    
    n = 2 * ((z_alpha + z_beta) / effect_size) ** 2
    return int(math.ceil(n))

# Detect 5% quality improvement: ~3,140 samples per variant
# Detect 10% quality improvement: ~786 samples per variant
# At 100 requests/hour: 1-2 days for a typical A/B test

5. Continuous Quality Monitoring

After deployment, quality can drift without model changes (data distribution shifts, new user patterns):

class QualityMonitor:
    """Continuous quality monitoring on production traffic."""
    
    async def monitor(self, sample_rate=0.02):
        """Sample 2% of production traffic for quality assessment."""
        while True:
            recent = await self.get_recent_requests(minutes=60, sample_rate=sample_rate)
            
            # Automated quality scoring
            scores = []
            for request in recent:
                score = await self.llm_judge.evaluate(
                    request.prompt, request.response
                )
                scores.append(score)
            
            avg_quality = mean(scores)
            quality_p10 = percentile(scores, 10)  # Bottom 10% quality
            
            # Compare against rolling baseline
            baseline = await self.get_baseline_quality(window_days=7)
            
            metrics.record("quality.avg", avg_quality)
            metrics.record("quality.p10", quality_p10)
            metrics.record("quality.baseline", baseline)
            
            if avg_quality < baseline - 0.3:
                await alert(
                    severity="warning",
                    summary=f"Quality dropped: {avg_quality:.2f} vs baseline {baseline:.2f}",
                )
            
            await asyncio.sleep(3600)  # Check every hour

Quality Dashboard Panels

Row 1: Deployment Status
  ├── Current model version (text)
  ├── Canary status (progress bar)
  └── Last rollback (timestamp + reason)

Row 2: Quality Metrics
  ├── Average quality score (line, 7-day)
  ├── Quality distribution (histogram)
  └── Quality by category (heatmap: topic × score)

Row 3: User Signals
  ├── Thumbs up/down ratio (line, 7-day)
  ├── Retry rate (line) — users resending same query
  └── Conversation length (line) — getting shorter = good?

Row 4: Safety
  ├── Guardrail trigger rate (line)
  ├── Safety violations (counter, should be ~0)
  └── False refusal rate (line, from sampled eval)

For offline evaluation before deployment, see Offline Evaluation. For alerting on quality drops, see Alerting.

Self-Check

  • Implement shadow mode for zero-risk comparison of new and production models
  • Design a multi-stage canary rollout with automatic rollback conditions
  • Calculate minimum sample size for an A/B test detecting 5% quality improvement
  • Set up continuous quality monitoring with LLM-as-judge on sampled traffic
  • Build a quality dashboard with deployment status, quality trends, and user signals

References