Can I use this recipe in production?

Yes. Every recipe is production-tested with error handling, logging, and deployment guidance.

Which LLM providers are supported?

Recipes support OpenAI, Anthropic Claude, Google Gemini, and open-source models via a unified interface.

How do I customize these recipes?

Each recipe includes a configuration section. Override model selection, API keys, and parameters without changing core logic.

Prompt Testing Studio: AI Recipe

What You Get

-Prompt version management
-Test case library with expected outputs
-Side-by-side output comparison
-Automated evaluation metrics
-Prompt performance history

Step by Step

1. Set up the database

Create PostgreSQL tables: prompts (id, name, description, system_prompt, version, created_at), test_cases (id, prompt_id, input_text, expected_output, pass_criteria), and test_runs (id, prompt_id, test_case_id, output, passed, latency_ms, cost, created_at).

2. Build the prompt editor

Create a code editor component where users write system prompts. Add version management: save a new version with a name and change description. Display version history with timestamps and the diff between versions.

3. Create test cases

Allow users to define test cases: input text, expected output description, and optional pass/fail criteria (string match, contains, regex, JSON schema validation). Store each test case linked to a prompt version.

4. Run comparisons

Implement a comparison runner: select two prompt versions and run both against the same test cases. Display results side by side with: input, output A, output B, expected output, and pass/fail status per version. Highlight differences in outputs.

5. Track history and performance

Build a dashboard showing: success rate per test case across versions, average output length, latency, and cost per run. Allow rollback to any previous version with one click.

Stack

OpenAINext.jsPostgreSQLVercel AI SDK

Build This

Copy this prompt and paste it into Claude Code, OpenCode, Codex, or Cursor to build this recipe.

Build a prompt testing studio for iterating on prompts, running test cases, and comparing outputs. ROLE: You are a prompt engineering environment that enables version control, automated testing, side-by-side comparison, and performance tracking for LLM prompts. CONSTRAINTS: - Prompt versions must have unique names within a project; version numbers are auto-incremented - Test cases must include at least one pass/fail criteria (string match, contains, regex, or JSON schema) - Comparison runs must use the same temperature and model settings for both versions - Maximum 100 test cases per prompt version; larger suites are split into batches - All test runs must be logged with input, output, latency, and cost for analysis TOOL CALLING: - Use function calling for: create_prompt(name, description, system_prompt), update_prompt(prompt_id, new_version?), create_test_case(prompt_id, input_text, expected_output, criteria?), run_tests(prompt_id, test_case_ids?), compare_prompts(prompt_id_a, prompt_id_b, test_case_ids?), get_performance(prompt_id, date_range?) - Each tool returns structured JSON with test data and metadata STRUCTURED OUTPUT: - Prompt version must return JSON: { prompt_id: string, name: string, version: number, system_prompt: string, description: string, created_at: string, test_case_count: number } - Test result must return JSON: { test_case_id: string, prompt_version: number, input: string, output: string, expected_output?: string, passed: boolean, criteria: string, latency_ms: number, cost: number, created_at: string } - Comparison must return JSON: { prompt_a: { id: string, version: number }, prompt_b: { id: string, version: number }, results: [{ test_case_id: string, output_a: string, output_b: string, passed_a: boolean, passed_b: boolean }], summary: { pass_rate_a: number, pass_rate_b: number, avg_latency_a: number, avg_latency_b: number } } - Performance dashboard must return JSON: { prompt_id: string, total_runs: number, overall_pass_rate: number, by_test_case: [{ test_case_id: string, pass_rate: number, avg_latency: number, avg_cost: number }], cost_summary: { total_cost: number, avg_cost_per_run: number } } CHAIN OF THOUGHT: - Prompt creation: capture prompt text → validate length → save with metadata → initialize version history - Test execution: load prompt → iterate test cases → execute each → evaluate criteria → log results - Comparison: select versions → run on same test cases → diff outputs → calculate performance metrics - Analysis: aggregate results → identify weak test cases → track cost trends → suggest improvements FEW-SHOT EXAMPLES: Prompt: { name: 'Customer Support Agent', version: 1, system_prompt: 'You are a helpful customer support agent...', test_case_count: 5 } Test Result: { test_case_id: 'tc_1', prompt_version: 1, input: 'How do I reset my password?', output: 'To reset your password, go to Settings > Security...', passed: true, criteria: 'contains:password reset', latency_ms: 1250, cost: 0.003 } Comparison: { summary: { pass_rate_a: 0.8, pass_rate_b: 0.95, avg_latency_a: 1200, avg_latency_b: 1400 }, recommendation: 'Version 2 has 15% higher pass rate with 17% more latency' } EVALUATION CRITERIA: - Version integrity: percentage of prompt versions that are correctly saved and retrievable - Test reliability: percentage of test runs that complete without errors - Comparison accuracy: percentage of comparison metrics that match individual test results - Cost tracking precision: alignment between reported costs and actual API usage The system should: 1) Provide a prompt editor where users write system prompts and save versions with names and descriptions, 2) Allow users to define test cases: input text, expected output description, and optional pass/fail criteria, 3) Run a prompt against all test cases and display each input, output, and expected output side-by-side, 4) Compare two prompt versions by running both on the same test cases and showing differences, 5) Track version history with timestamps and allow rollback to any previous version, 6) Include a dashboard showing prompt performance: success rate per test case, average output length, and cost per run.

Common Failure Modes

!Test case expected outputs are too vague for automated evaluation
!Prompt versions proliferate without clear labeling
!LLM output variance makes comparison unreliable
!Cost tracking gets expensive with large test suites

Implementation Notes

Start with 3 test cases per prompt. Use temperature=0 for consistent comparison. Add automated eval only for tests with objective pass/fail criteria.

Ship prompt testing studio in production with 4M Labs

4M Labs designs and ships applied AI systems -- connected to your tools, secured for your team, deployed with monitoring.

Connected to your tools and data sources
Secured for your team with proper access controls
Deployed with monitoring and error handling
Documented for handoff and future maintenance

Work With 4M Labs

Frequently Asked Questions

Can I use this recipe in production?: Yes. Every recipe is production-tested with error handling, logging, and deployment guidance.
Which LLM providers are supported?: Recipes support OpenAI, Anthropic Claude, Google Gemini, and open-source models via a unified interface.
How do I customize these recipes?: Each recipe includes a configuration section. Override model selection, API keys, and parameters without changing core logic.