Back to Patterns

Prompt Versioning and Evals

Prompt Patterns

Summary

Prompt versioning and evals apply software engineering practices -- version control, regression testing, and rollback -- to prompt management. Every prompt change is tracked, tested against a suite of evaluation cases, and deployed only when it passes quality gates.

How it works

  1. Version each prompt -- store every prompt variant with a unique version ID and changelog.
  2. Define test cases -- create a suite of input-output pairs that represent expected behavior.
  3. Run evals on changes -- every prompt update triggers automated evaluation against the test suite.
  4. Track regressions -- compare eval results to the previous version to catch quality drops.

What to track

  • Success rate: Percentage of test cases that produce acceptable outputs.
  • Output quality: Human or LLM-as-judge scores for helpfulness, accuracy, tone.
  • Cost per call: Token usage changes introduced by the new prompt.
  • Latency: Time-to-first-token and total generation time.

Rollback

When a prompt change causes quality regression, revert to the previous version immediately. Keep the failed version and its eval results for post-mortem analysis.

Build This Pattern

Copy this prompt and paste it into Claude Code, OpenCode, Codex, or Cursor to implement this pattern.

Build a prompt versioning and eval system for production prompts. ROLE: You are a prompt lifecycle management system that versions prompts, runs regression tests, and tracks performance across versions. CONSTRAINTS: - Each prompt version has unique ID, hash, timestamp, and associated test suite - Store versions in database with metadata (author, description, change reason) - Regression detection: compare eval scores across versions - Always run full eval suite before promoting to production - Support prompt rollback with behavior verification TOOL CALLING: - Use function calling for: create_version(prompt_data, description?), promote_version(version_id, environment?), rollback_version(version_id), run_eval(version_id, test_suite_id?), compare_versions(version_a, version_b) - Each tool returns structured JSON with version data and metadata STRUCTURED OUTPUT: - Prompt version must return JSON: { version_id: string, name: string, hash: string, content: string, description: string, author: string, created_at: string, status: 'draft' | 'testing' | 'production' | 'archived', test_suite_id?: string } - Eval result must return JSON: { version_id: string, test_suite_id: string, total_tests: number, passed: number, failed: number, score: number, regressions: [{ test_name: string, expected: string, actual: string, severity: string }], execution_time_ms: number } - Version comparison must return JSON: { version_a: string, version_b: string, content_diff: string, eval_diff: { score_a: number, score_b: number, improvement: number }, recommendation: 'promote' | 'keep' | 'rollback' } - Rollback result must return JSON: { rolled_back_from: string, rolled_back_to: string, verified: boolean, behavior_match: boolean, verification_tests: number } CHAIN OF THOUGHT: - Versioning: create new version → generate hash → store with metadata → link to test suite - Evaluation: run test suite → compare to baseline → identify regressions → calculate scores - Promotion: verify all tests pass → compare to current production → approve or reject - Rollback: select target version → restore content → run verification tests → confirm behavior match FEW-SHOT EXAMPLES: Version: { version_id: 'v_123', name: 'Customer Support v2.1', hash: 'abc123', status: 'testing', created_at: '2025-01-15T10:00:00Z' } Eval: { version_id: 'v_123', total_tests: 25, passed: 24, failed: 1, score: 0.96, regressions: [{ test_name: 'tone_check', expected: 'professional', actual: 'casual', severity: 'medium' }] } Comparison: { score_a: 0.92, score_b: 0.96, improvement: 0.04, recommendation: 'promote' } Rollback: { rolled_back_from: 'v_124', rolled_back_to: 'v_122', verified: true, behavior_match: true, verification_tests: 25 } EVALUATION CRITERIA: - Version integrity: percentage of versions correctly stored and retrievable - Regression detection: percentage of regressions correctly identified - Promotion safety: percentage of promotions that don't introduce regressions - Rollback reliability: percentage of rollbacks that restore previous behavior The system should: 1) Each prompt version has ID, hash, timestamp, and associated test suite, 2) Store versions in database with metadata, 3) Detect regressions by comparing eval scores across versions, 4) Handle prompt rollback, 5) Handle version conflicts in team environments, 6) Always run full eval suite before promoting prompt to production, 7) Verify that rollback correctly restores previous behavior.