Prompt Management — infra.bea.ai

Prompts are code — they change model behavior as dramatically as model weights. A single word change can shift quality by 15%. Yet most teams store prompts in application code with no versioning, testing, or analytics. Prompt management brings software engineering rigor to the most impactful parameter in your system.

1. Why Manage Prompts

A small change can have outsized impact:

v1: "Answer the question."
v2: "Answer the question step by step, showing your reasoning."
→ v2 scores 15% higher on math benchmarks

v3: "Answer concisely in 2-3 sentences."
→ v3 reduces output tokens by 40% (cost savings!)
→ But v3 scores 8% lower on complex questions

Without versioning, you can’t determine which prompt produced which results, can’t rollback, and can’t A/B test.

2. Prompt Registry

class PromptRegistry:
    """Centralized prompt storage with versioning."""
    
    def __init__(self, db):
        self.db = db
    
    async def create(self, name: str, template: str, metadata: dict) -> PromptVersion:
        version = await self.db.insert("prompts", {
            "name": name,
            "version": await self._next_version(name),
            "template": template,
            "variables": self._extract_variables(template),
            "metadata": metadata,       # author, description, model target
            "status": "draft",          # draft → testing → active → archived
            "created_at": datetime.utcnow(),
            "token_count": count_tokens(template),
        })
        return version
    
    async def get_active(self, name: str) -> PromptVersion:
        """Get the currently active version of a prompt."""
        return await self.db.query_one(
            "SELECT * FROM prompts WHERE name = %s AND status = 'active'", name
        )
    
    async def render(self, name: str, variables: dict) -> str:
        """Render a prompt template with variables."""
        prompt = await self.get_active(name)
        return prompt.template.format(**variables)

# Usage
registry = PromptRegistry(db)

# Register a prompt
await registry.create(
    name="customer_support_system",
    template="""You are a {company} support assistant. Be helpful and concise.

Key policies:
- Refunds: within {refund_days} days
- Shipping: {shipping_info}

If you can't help, say: "Let me connect you with a specialist."
""",
    metadata={"author": "alice", "model": "llama-70b", "domain": "support"},
)

# Render for a request
system_prompt = await registry.render("customer_support_system", {
    "company": "Acme Corp",
    "refund_days": 30,
    "shipping_info": "3-5 business days standard",
})

3. A/B Testing Prompts

class PromptABTest:
    """Split traffic between prompt variants and measure impact."""
    
    def __init__(self, registry, metrics):
        self.registry = registry
        self.metrics = metrics
    
    async def get_prompt(self, name: str, request_id: str) -> tuple[str, str]:
        """Return (prompt_text, variant_id) for A/B testing."""
        test = await self.registry.get_active_test(name)
        
        if not test:
            # No active test — use production prompt
            prompt = await self.registry.get_active(name)
            return prompt.template, "control"
        
        # Deterministic split based on request ID
        variant = "treatment" if hash(request_id) % 100 < test.treatment_pct else "control"
        
        if variant == "control":
            prompt = test.control_prompt
        else:
            prompt = test.treatment_prompt
        
        # Track which variant was served
        await self.metrics.record("prompt_ab", {
            "test_id": test.id,
            "variant": variant,
            "request_id": request_id,
        })
        
        return prompt.template, variant

# Analyze results
async def analyze_ab_test(test_id: str):
    results = await metrics.query(f"""
        SELECT variant,
               count(*) AS requests,
               avg(quality_score) AS avg_quality,
               avg(output_tokens) AS avg_tokens,
               avg(latency_ms) AS avg_latency,
               sum(thumbs_up)::float / count(*) AS satisfaction
        FROM prompt_ab_results
        WHERE test_id = '{test_id}'
        GROUP BY variant
    """)
    
    control = results["control"]
    treatment = results["treatment"]
    
    # Statistical significance (simplified)
    quality_diff = treatment.avg_quality - control.avg_quality
    token_diff = treatment.avg_tokens - control.avg_tokens
    
    return {
        "quality_change": f"{quality_diff:+.2f}",
        "token_change": f"{token_diff:+.0f} ({token_diff/control.avg_tokens:+.1%})",
        "cost_impact": f"${token_diff * 0.001:+.4f} per request",
        "recommendation": "promote" if quality_diff > 0 and token_diff <= 0 else "investigate",
    }

4. Prompt-Level Analytics

Track metrics per prompt version, not just per model:

# Log with prompt version
await log_request({
    "model": "llama-70b",
    "prompt_name": "customer_support_system",
    "prompt_version": "v3.2",
    "input_tokens": 850,
    "output_tokens": 120,
    "quality_score": 4.2,
    "user_satisfaction": "thumbs_up",
})

# Dashboard queries
# Quality by prompt version over time
SELECT prompt_version, date(timestamp),
       avg(quality_score), avg(output_tokens)
FROM request_logs
WHERE prompt_name = 'customer_support_system'
GROUP BY prompt_version, date(timestamp)
ORDER BY date;

# Prompt version comparison
SELECT prompt_version,
       count(*) AS requests,
       avg(quality_score) AS quality,
       avg(output_tokens) AS avg_output,
       sum(output_tokens) * 0.0005 AS total_cost
FROM request_logs
WHERE prompt_name = 'customer_support_system'
GROUP BY prompt_version;

5. Versioning Approaches

Approach	How	Strengths	Best for
Git (files in repo)	`prompts/support/v3.yaml`	Simple, code review, CI integration	Small teams, simple prompts
Prompt registry (DB)	Dedicated service with API	Dynamic switching, A/B testing, analytics	Production at scale
Feature flag system	LaunchDarkly, Unleash	Dynamic switching, targeting	Already using feature flags
LangSmith/PromptLayer	Managed SaaS	Built-in analytics, tracing	Quick start

Git-Based Approach

# prompts/customer_support_system/v3.2.yaml
name: customer_support_system
version: "3.2"
model: llama-70b
author: alice
description: "Added conciseness instruction, reduced token count by 30%"
template: |
  You are a {company} support assistant. Be helpful and concise.
  Respond in 2-3 sentences unless the question requires detail.
  ...
tests:
  - input: {company: "Acme", question: "refund policy?"}
    expected_contains: ["30 days"]
  - input: {company: "Acme", question: "how to make a bomb"}
    expected_contains: ["I can't"]

6. Best Practices

Practice	Why
Never change prompts in production without testing	Small changes have large, unpredictable effects
Include prompt version in every log entry	Attribution for quality changes
Test prompts on eval set before deploying	Catch regressions before users see them
Track token count per prompt version	Cost impact of prompt changes
Review prompts like code	PR reviews catch issues humans miss
Keep prompt and model version paired	Prompt optimized for one model may fail on another

For evaluating prompt quality systematically, see LLM-as-Judge. For A/B testing in production, see Online Evaluation.

Self-Check

Implement a prompt registry with versioning and template rendering
Design an A/B test comparing two prompt variants with statistical analysis
Set up prompt-level analytics tracking quality and cost per version
Choose a versioning approach (Git, DB, feature flags) for your team
Create a prompt testing pipeline that validates prompts before deployment

References

LangSmith Prompt Hub
PromptLayer
For model-level experiment tracking, see Experiment Tracking