Evaluator-Optimizer

Build an evaluator-optimizer loop for iterative LLM output refinement. ROLE: You are a quality optimization system that generates initial outputs, evaluates them against criteria, and iteratively improves until quality thresholds are met. CONSTRAINTS: - Maximum 3 iterations per optimization loop; early exit when all criteria pass - Quality criteria must be defined as rules with weights and pass/fail thresholds - Loop must detect stagnation: no improvement over 2 consecutive iterations triggers exit - Each iteration must log output version, feedback, and criterion scores - Final output must include full iteration history for analysis TOOL CALLING: - Use function calling for: generate_initial_output(task, criteria[]), evaluate_output(output, criteria[]), optimize_output(output, feedback, criteria[]), get_optimization_history(loop_id?) - Each tool returns structured JSON with optimization data and metadata STRUCTURED OUTPUT: - Generation must return JSON: { output_id: string, content: string, iteration: number, generation_time_ms: number } - Evaluation must return JSON: { output_id: string, criteria: [{ name: string, weight: number, score: number, passed: boolean, feedback?: string }], overall_score: number, passed: boolean, improvement_from_previous?: number } - Optimization history must return JSON: { loop_id: string, iterations: [{ output_id: string, content: string, evaluation: { overall_score: number, passed: boolean }, feedback?: string }], final_output_id: string, total_time_ms: number, improvement_trajectory: number[] } CHAIN OF THOUGHT: - Initial generation: analyze task → understand criteria → generate baseline output → log - Evaluation: assess against each criterion → calculate weighted score → identify weaknesses → provide feedback - Optimization: apply feedback → regenerate output → compare to previous → track improvement - Termination: check pass criteria → detect stagnation → select best version → return with history FEW-SHOT EXAMPLES: Task: 'Write a product description for a wireless headphone' Criteria: [{ name: 'clarity', weight: 0.3, score: 8, passed: true }, { name: 'persuasiveness', weight: 0.4, score: 6, passed: false, feedback: 'Add more emotional appeal' }, { name: 'technical_accuracy', weight: 0.3, score: 9, passed: true }] Overall: { overall_score: 7.5, passed: false } Iteration 2: { overall_score: 8.8, passed: true, improvement_from_previous: 1.3 } History: { improvement_trajectory: [7.5, 8.8] } EVALUATION CRITERIA: - Iteration efficiency: average number of iterations to reach pass criteria - Quality improvement: percentage improvement from first to final iteration - Stagnation detection: percentage of loops that correctly exit on no improvement - History completeness: percentage of iterations with complete logging The system should: 1) Implement generator module that creates initial output, 2) Implement evaluator module that reviews output against quality criteria and provides specific, actionable feedback, 3) Generator uses feedback to produce improved version, 4) Loop continues until evaluator passes output or max iterations reached (configurable default 3), 5) Define quality criteria as array of rules with weights and pass/fail thresholds, 6) Track full iteration history including each output version, feedback, and criterion scores, 7) Detect loops where output stops improving by comparing similarity scores, 8) If no improvement over 2 consecutive iterations, return best version and break, 9) Handle evaluator failure by using last valid evaluation, 10) Handle single-criterion optimization efficiently by bailing early when met, 11) Support criteria that are pass/fail versus scored (1-10).