Back to Patterns

Trace and Eval Flywheel

Agent Patterns

Summary

The trace and eval flywheel is a continuous improvement loop for agent quality. Every agent execution is traced, graded against quality criteria, and failures are collected into a regression dataset. That dataset drives prompt and logic improvements, and the cycle repeats.

How it works

  1. Trace -- capture every agent invocation: input, output, tool calls, intermediate state, latency.
  2. Grade -- evaluate each trace against quality criteria using automated graders or human review.
  3. Collect failures -- aggregate failed or low-quality traces into a structured regression dataset.
  4. Improve -- analyze the dataset, update prompts or logic, and deploy the fix.
  5. Repeat -- the improved agent generates new traces, and the cycle continues.

Components

  • Tracer: Instrumentation layer that records every step of agent execution.
  • Grader: Automated evaluator (LLM-as-judge, metric computation, or rule-based checks).
  • Regression dataset: Versioned collection of failure cases used to prevent recurring issues.
  • Improvement engine: Tooling to analyze failures, suggest fixes, and validate improvements.

Metrics

  • Improvement rate: Percentage of regression cases that pass after each improvement cycle.
  • Regression count: Number of previously-passing cases that fail after a change.
  • Grade distribution: Breakdown of traces by quality score across the system.

Build This Pattern

Copy this prompt and paste it into Claude Code, OpenCode, Codex, or Cursor to implement this pattern.

Build a trace and eval flywheel system for continuous improvement. ROLE: You are an observability and quality system that captures traces, grades them against criteria, builds regression datasets, and iteratively improves prompts and tools. CONSTRAINTS: - Every agent interaction must be traced (inputs, outputs, tool calls, latencies) - Grading criteria must be defined before traces are collected - Regression datasets must include at least 10 examples per failure category - Improvement cycles must be measured; track improvement rate across iterations - Trace storage: maximum 1GB per project; oldest traces are archived TOOL CALLING: - Use function calling for: capture_trace(interaction_data), grade_trace(trace_id, criteria[]), add_to_regression_dataset(failure_data), run_eval(dataset_id, prompt_version?), get_improvement_metrics(project_id?) - Each tool returns structured JSON with trace data and metadata STRUCTURED OUTPUT: - Trace must return JSON: { trace_id: string, interaction_id: string, inputs: Record<string, any>, outputs: Record<string, any>, tool_calls: [{ name: string, latency_ms: number, status: string }], total_latency_ms: number, captured_at: string } - Grade result must return JSON: { trace_id: string, grades: [{ criterion: string, score: number, passed: boolean, feedback?: string }], overall_score: number, passed: boolean } - Regression dataset must return JSON: { dataset_id: string, category: string, examples: [{ input: string, expected_output: string, failure_type: string }], total_examples: number, created_at: string } - Improvement metrics must return JSON: { project_id: string, total_traces: number, graded_traces: number, regression_datasets: number, improvement_rate: number, latest_eval_score: number, previous_eval_score: number } CHAIN OF THOUGHT: - Capture: instrument agent → collect interaction data → store trace with metadata - Grading: load grading criteria → assess trace against criteria → assign scores → identify failures - Dataset building: collect failures → categorize by type → create regression examples → store dataset - Improvement: run eval with new prompts → compare scores → iterate on weak areas → measure progress FEW-SHOT EXAMPLES: Trace: { trace_id: 'tr_123', inputs: { query: 'Summarize this document' }, outputs: { summary: '...' }, tool_calls: [{ name: 'search_web', latency_ms: 1200, status: 'success' }], total_latency_ms: 3500 } Grade: { trace_id: 'tr_123', grades: [{ criterion: 'accuracy', score: 8, passed: true }, { criterion: 'completeness', score: 6, passed: false, feedback: 'Missing key points from section 3' }], overall_score: 7, passed: false } Dataset: { dataset_id: 'ds_456', category: 'incomplete_summaries', examples: [{ input: 'Long document...', expected_output: 'Complete summary with all sections...', failure_type: 'missing_sections' }], total_examples: 15 } Metrics: { improvement_rate: 0.12, latest_eval_score: 8.2, previous_eval_score: 7.3 } EVALUATION CRITERIA: - Trace completeness: percentage of interactions that are fully captured - Grading accuracy: correlation between automated grades and human evaluation - Dataset quality: percentage of regression examples that are representative - Improvement measurement: accuracy of improvement rate calculations The system should: 1) Capture traces of every agent interaction (inputs, outputs, tool calls, latencies), 2) Grade traces against success criteria, 3) Collect failures into regression dataset, 4) Use dataset to improve prompts and re-run evals, 5) Handle trace corruption, oversized traces, 6) Handle low-grade-volume periods, grading disagreement, 7) Measure improvement rate across iterations, 8) Verify that system correctly identifies and captures failures.