Offline Evaluation

Offline evaluation measures model quality on standardized benchmarks and custom test suites before the model sees production traffic. It’s the primary quality gate in the model CI/CD pipeline — and the most common source of misleading conclusions when done poorly.

---

1. Standard Benchmarks

Benchmark	What it measures	Format	Typical score (70B)	Tokens/eval
MMLU	Knowledge across 57 subjects	5-shot multiple choice	78-85%	~2M
HumanEval	Code generation (Python)	Zero-shot code completion	60-80% (pass@1)	~200K
GSM8K	Grade school math reasoning	8-shot chain-of-thought	80-92%	~500K
MATH	Competition math	4-shot formal proof	40-65%	~1M
MT-Bench	Multi-turn conversation quality	LLM-as-judge scoring	8.0-9.0/10	~100K
IFEval	Instruction following precision	Format compliance	70-85%	~300K
GPQA	PhD-level science questions	0-shot multiple choice	35-55%	~100K
ARC-Challenge	Grade-school science reasoning	25-shot multiple choice	85-95%	~300K

Total eval cost: Full suite on 70B model ≈ 1-2 GPU-hours on H100 (with vLLM batching).

---

2. lm-evaluation-harness

The standard tool for reproducible offline evaluation:

# Basic evaluation
lm_eval --model vllm \
    --model_args pretrained=meta-llama/Llama-3.1-70B,tensor_parallel_size=4 \
    --tasks mmlu,humaneval,gsm8k,ifeval \
    --batch_size auto \
    --output_path results/

# Compare two models
lm_eval --model vllm \
    --model_args pretrained=./checkpoints/v2.3,tensor_parallel_size=4 \
    --tasks mmlu,humaneval,gsm8k \
    --batch_size auto \
    --output_path results/v2.3/

# Custom few-shot configuration
lm_eval --model vllm \
    --model_args pretrained=./checkpoints/v2.3 \
    --tasks mmlu \
    --num_fewshot 5 \
    --output_path results/

Key lm-eval Features

Feature	How to use	Why it matters
200+ built-in tasks	--tasks mmlu,humaneval	Standardized metrics
vLLM backend	--model vllm	Fast batched evaluation
Few-shot control	--num_fewshot 5	Reproducible prompting
Output logging	--log_samples	Debug wrong answers
Custom tasks	YAML task configs	Domain-specific evaluation

---

3. Custom Benchmark Suites

Standard benchmarks miss domain-specific quality. Build custom suites for your application:

# Custom evaluation suite configuration
eval_suite = {
    "domain_knowledge": {
        "dataset": "internal/medical_qa_v3.jsonl",
        "metric": "accuracy",
        "threshold": 0.85,
        "description": "Medical Q&A from real clinical scenarios",
    },
    "format_compliance": {
        "dataset": "internal/json_format_test.jsonl",
        "metric": "valid_json_rate",
        "threshold": 0.95,
        "description": "Output must be valid JSON matching schema",
    },
    "safety": {
        "dataset": "internal/safety_prompts_v2.jsonl",
        "metric": "refusal_rate",
        "threshold": 0.95,
        "description": "Must refuse harmful requests",
    },
    "regression": {
        "dataset": "internal/golden_examples.jsonl",
        "metric": "semantic_similarity",
        "threshold": 0.90,
        "description": "Key examples must produce similar answers to previous version",
    },
}

# Custom task for lm-eval-harness
# tasks/medical_qa.yaml
task: medical_qa
dataset_path: json
dataset_name: null
dataset_kwargs:
  data_files: "internal/medical_qa_v3.jsonl"
output_type: generate_until
generation_kwargs:
  until: ["\n\n"]
  max_gen_toks: 256
metric_list:
  - metric: exact_match
    aggregation: mean

Building Good Custom Benchmarks

Principle	Implementation
Representative	Sample from real production queries
Non-contaminated	Ensure examples aren’t in training data
Versioned	Tag dataset with version, lock it
Diverse	Cover edge cases, not just easy examples
Human-validated	Expert review of ground truth answers
Regularly updated	Add new failure modes as they’re discovered

---

4. Contamination Detection

A model that scored 95% on MMLU might have seen MMLU questions during training:

def check_contamination(model, benchmark_examples, n_prefix_tokens=50):
    """Check if model can complete benchmark examples from memory."""
    contaminated = 0
    
    for example in benchmark_examples:
        # Give the model the first N tokens
        prefix = tokenize(example["question"])[:n_prefix_tokens]
        continuation = model.generate(prefix, max_tokens=200)
        
        # If the model reproduces the answer verbatim, it's memorized
        if example["answer"] in continuation:
            contaminated += 1
    
    contamination_rate = contaminated / len(benchmark_examples)
    if contamination_rate > 0.05:
        logger.warning(f"Possible contamination: {contamination_rate:.1%} of examples memorized")
    
    return contamination_rate

Mitigation: Hold out a private evaluation set that is never published or included in any training data.

---

5. Evaluation Infrastructure

For production-grade evaluation at scale:

class EvalPipeline:
    """Automated evaluation pipeline for model CI/CD."""
    
    def __init__(self, gpu_pool, registry, alerter):
        self.gpu_pool = gpu_pool
        self.registry = registry
        self.alerter = alerter
    
    async def evaluate(self, checkpoint_path: str, model_name: str) -> EvalReport:
        # 1. Allocate GPU for evaluation
        gpu = await self.gpu_pool.allocate(tp_size=4, timeout=300)
        
        try:
            # 2. Run standard benchmarks
            standard = await self.run_lm_eval(
                checkpoint_path, gpu,
                tasks=["mmlu", "humaneval", "gsm8k", "ifeval"],
            )
            
            # 3. Run custom suite
            custom = await self.run_custom_suite(checkpoint_path, gpu)
            
            # 4. Compare against production baseline
            production = await self.registry.get_production(model_name)
            regression = self.check_regression(standard, production.eval_scores)
            
            # 5. Generate report
            report = EvalReport(
                standard_scores=standard,
                custom_scores=custom,
                regressions=regression,
                passed=len(regression.failures) == 0,
            )
            
            if not report.passed:
                await self.alerter.notify(
                    f"Eval failed for {model_name}: {regression.failures}"
                )
            
            return report
        finally:
            await self.gpu_pool.release(gpu)

Compute budget: Plan for eval costs in your GPU allocation:

• Standard suite (MMLU+HumanEval+GSM8K): ~1-2 GPU-hours
• Full suite with custom benchmarks: ~2-4 GPU-hours
• Evaluating every checkpoint during training: budget 5-10% of training compute for eval

For using LLMs to evaluate LLMs, see LLM-as-Judge. For production quality monitoring, see Online Evaluation.

---

Self-Check

• Run lm-evaluation-harness with 3+ benchmarks on a model
• Design a custom evaluation suite for a domain-specific application
• Implement contamination detection for standard benchmarks
• Set up an automated evaluation pipeline triggered by new checkpoints
• Define pass/fail thresholds and regression tolerances for CI/CD gates

---

References

• lm-evaluation-harness
• MMLU (Hendrycks et al., 2021)
• HumanEval (Chen et al., 2021)
• Contamination in LLM Benchmarks (Oren et al., 2023)