Human Evaluation

Human evaluation is the gold standard for measuring model quality — it captures nuances that automated metrics miss. But it’s expensive ($0.10-2.00 per evaluation), slow (days to weeks), and unreliable without careful quality control.

---

1. Evaluation Types

Type	Task	Throughput	Cost/item	Use case
Absolute scoring	Rate response 1-5 on criteria	200-400/hr	$0.10-0.25	Quality monitoring
Pairwise comparison	“Which response is better?”	100-200/hr	$0.20-0.50	RLHF preference data, model comparison
Error annotation	Mark specific errors in output	50-100/hr	$0.50-1.00	Debugging, targeted fine-tuning
Red-teaming	Try to elicit harmful outputs	20-50/hr	$1.00-5.00	Safety evaluation
Factual verification	Check claims against sources	30-60/hr	$0.50-2.00	Hallucination detection

---

2. Annotation Platform Setup

Tools

Platform	Type	Strengths	Best for
Argilla	Open-source	LLM-focused, HuggingFace integration	AI teams, preference data
Label Studio	Open-source	Flexible, many data types	General annotation
Scale AI	Managed service	Large workforce, quality guarantees	High-volume, production
Surge AI	Managed service	AI-specialized annotators	RLHF, quality-sensitive
Prolific	Crowdsource	Academic-quality participants	Research, diverse demographics

Custom Annotation Interface

# Argilla: set up pairwise comparison task
import argilla as rg

dataset = rg.FeedbackDataset(
    fields=[
        rg.TextField(name="prompt", title="User Prompt"),
        rg.TextField(name="response_a", title="Response A"),
        rg.TextField(name="response_b", title="Response B"),
    ],
    questions=[
        rg.RatingQuestion(
            name="preference",
            title="Which response is better?",
            description="1=A much better, 3=tie, 5=B much better",
            values=[1, 2, 3, 4, 5],
        ),
        rg.TextQuestion(
            name="reasoning",
            title="Why?",
            description="Brief explanation of your choice",
            required=True,
        ),
    ],
    guidelines="""
    Compare the two responses on: helpfulness, accuracy, and clarity.
    - Rate 1 if Response A is clearly better
    - Rate 5 if Response B is clearly better
    - Rate 3 if they are roughly equal
    Ignore minor formatting differences. Focus on substance.
    """,
)

---

3. Quality Control

Without quality control, annotations are noisy and unreliable:

Inter-Annotator Agreement

from sklearn.metrics import cohen_kappa_score
import numpy as np

def measure_agreement(annotations: dict) -> dict:
    """Measure agreement across annotators on overlapping items."""
    # Get items rated by multiple annotators
    overlap_items = find_overlap(annotations)
    
    pairs = []
    for item_id in overlap_items:
        raters = annotations[item_id]
        for i in range(len(raters)):
            for j in range(i + 1, len(raters)):
                pairs.append((raters[i], raters[j]))
    
    scores_a = [p[0] for p in pairs]
    scores_b = [p[1] for p in pairs]
    
    kappa = cohen_kappa_score(scores_a, scores_b)
    exact_agree = sum(a == b for a, b in pairs) / len(pairs)
    close_agree = sum(abs(a - b) <= 1 for a, b in pairs) / len(pairs)
    
    return {
        "cohens_kappa": kappa,       # > 0.7 good, > 0.8 excellent
        "exact_agreement": exact_agree,
        "close_agreement": close_agree,  # Within 1 point
        "n_overlap": len(overlap_items),
    }

Quality Control Mechanisms

Mechanism	How it works	Purpose
Calibration questions	Hidden known-answer items (~10% of tasks)	Detect inattentive annotators
Inter-annotator overlap	15-20% of items rated by 2+ annotators	Measure reliability
Gold standards	Expert-labeled reference items	Benchmark annotator accuracy
Attention checks	Obvious items (“select option A”)	Catch random clicking
Reasoning required	Must explain every rating	Forces deliberation
Regular audits	Review random 5% of annotations	Catch systematic errors

# Annotator quality monitoring
async def monitor_annotator(annotator_id: str) -> AnnotatorQuality:
    recent = await get_recent_annotations(annotator_id, n=100)
    
    # Check calibration accuracy
    calibration_items = [a for a in recent if a.is_calibration]
    calibration_accuracy = mean(a.correct for a in calibration_items)
    
    # Check agreement with other annotators
    overlap = await get_overlap_agreement(annotator_id)
    
    # Check for suspicious patterns
    all_same = len(set(a.score for a in recent)) == 1  # Always rates the same
    too_fast = mean(a.time_seconds for a in recent) < 10  # Suspiciously fast
    
    quality = AnnotatorQuality(
        accuracy=calibration_accuracy,      # Target: > 0.85
        agreement=overlap.kappa,            # Target: > 0.65
        suspicious=all_same or too_fast,
    )
    
    if quality.accuracy < 0.70 or quality.suspicious:
        await flag_annotator(annotator_id, reason="Low quality")
    
    return quality

---

4. Collecting RLHF Preference Data

# Generate comparison pairs for annotators
async def create_comparison_batch(prompts, model_a, model_b, batch_size=500):
    """Generate response pairs for human preference annotation."""
    pairs = []
    for prompt in prompts[:batch_size]:
        resp_a = await model_a.generate(prompt, temperature=0.7)
        resp_b = await model_b.generate(prompt, temperature=0.7)
        
        # Randomize order to prevent position bias
        if random.random() > 0.5:
            pairs.append({"prompt": prompt, "response_a": resp_a, "response_b": resp_b,
                         "model_a": model_a.name, "model_b": model_b.name})
        else:
            pairs.append({"prompt": prompt, "response_a": resp_b, "response_b": resp_a,
                         "model_a": model_b.name, "model_b": model_a.name})
    
    return pairs

# Convert annotations to reward model training data
def annotations_to_preference_pairs(annotations):
    """Convert human annotations to (chosen, rejected) pairs."""
    pairs = []
    for ann in annotations:
        if ann.preference < 3:  # A preferred
            pairs.append({
                "prompt": ann.prompt,
                "chosen": ann.response_a,
                "rejected": ann.response_b,
            })
        elif ann.preference > 3:  # B preferred
            pairs.append({
                "prompt": ann.prompt,
                "chosen": ann.response_b,
                "rejected": ann.response_a,
            })
        # Skip ties (preference == 3)
    return pairs

Scale Estimation

Data need	Comparisons needed	Annotator-days (3K/day)	Cost ($0.30/comparison)
Reward model (basic)	10K-50K	3-17 days	$3K-$15K
Reward model (good)	50K-200K	17-67 days	$15K-$60K
Reward model (production)	200K-1M	67-333 days	$60K-$300K

Cost optimization: Use LLM-as-judge for easy cases, human evaluation for ambiguous ones:

async def smart_annotation(prompt, response_a, response_b):
    """Use LLM judge first, human only for uncertain cases."""
    judge_result = await llm_judge.compare(prompt, response_a, response_b)
    
    if judge_result.confidence > 0.9:
        return judge_result  # High confidence — skip human
    else:
        return await send_to_human_annotator(prompt, response_a, response_b)
    
# Typically: 60-70% handled by LLM judge, 30-40% sent to humans
# Cost reduction: ~50-60%

---

5. Annotation Guidelines

The most underrated component — bad guidelines produce bad data:

# Pairwise Comparison Guidelines v2.1

## Task
Compare two AI responses and select the better one.

## Criteria (in order of importance)
1. **Accuracy**: Is the information correct? (Most important)
2. **Helpfulness**: Does it address the user's actual question?
3. **Safety**: Is it free from harmful content?
4. **Clarity**: Is it well-organized and easy to understand?
5. **Conciseness**: Does it avoid unnecessary repetition?

## Scoring
- 1: Response A is significantly better
- 2: Response A is slightly better
- 3: About equal (use sparingly)
- 4: Response B is slightly better
- 5: Response B is significantly better

## Examples
[Include 5-10 calibrated examples with explanations]

## Edge Cases
- If both responses are factually wrong, prefer the one that's less wrong
- If one is more helpful but the other is more accurate, prefer accuracy
- Formatting differences (bullet points vs paragraphs) are NOT a quality difference

For automated alternatives, see LLM-as-Judge. For safety-focused evaluation, see Safety Evaluation.

---

Self-Check

• Set up an annotation platform with pairwise comparison tasks
• Implement inter-annotator agreement measurement (Cohen’s κ)
• Design quality control with calibration items and attention checks
• Estimate cost and timeline for collecting 100K preference comparisons
• Write annotation guidelines with criteria, scoring, and calibration examples

---

References

• Argilla: Open-Source Data Curation for AI
• Scale AI Annotation Platform
• InstructGPT: Training Language Models to Follow Instructions (Ouyang et al., 2022)