Back to Patterns

Model Cascade Routing

Agent Patterns

Summary

Model routing and model cascading both send work to different models, but they make the decision at different points. A prompt-time router selects a model before generation. A response cascade starts with one model, evaluates its answer, and may pay to run another model. Both can reduce spend for a workload, but neither guarantees lower total cost or unchanged quality: measure the trade-off on your own requests.

Routing versus cascading

  • Prompt-time routing scores the request and chooses a model before an answer is generated. A learned router may estimate which model is more likely to win for that prompt; this score is not the cheaper model's confidence in an answer it has not produced yet.
  • Post-response cascading runs an initial model, then evaluates the generated answer. A task-specific verifier, a second model acting as judge, or a deterministic check can trigger another attempt when the answer fails the acceptance policy.

RouteLLM is an example of prompt-time routing. Its routers estimate the strong model's win rate for a prompt and compare that value with a configurable threshold. This differs from a cascade that judges a completed answer. See the RouteLLM paper for its preference-data approach.

How it works

  1. Define the decision point -- choose prompt-time routing when the request itself is enough to select a model; choose a response cascade when answer checks can detect a failure after generation.
  2. Set up tiers and policy -- record each model's cost, latency, and measured task quality. For a cascade, define answer acceptance checks and an escalation budget. For a learned router, choose the router score and threshold.
  3. Calibrate on representative traffic -- replay a held-out sample of real requests, including difficult and high-risk cases. Sweep thresholds or acceptance rules and inspect the resulting quality, cost, escalation, and latency. RouteLLM specifically advises calibration on queries representative of the workload because traffic mix changes routing rates.
  4. Escalate with a bounded policy -- in a cascade, send failed or uncertain answers to the next tier only when the expected benefit justifies another call. Set a maximum number of attempts and a clear fallback. In prompt-time routing, select the stronger model when the router's score crosses the calibrated cutoff.
  5. Monitor and recalibrate -- track outcomes as the request mix, model versions, prices, and latency change. Keep a safe route for tasks where an incorrect low-tier answer has a high cost.

Tiers

  • Lower-cost tier: Candidate for constrained, well-evaluated tasks such as classification or extraction. Confirm quality on your data before routing production work here.
  • Default tier: The model that meets the workload's normal quality and latency requirements.
  • Fallback tier: A more capable model for requests that fail validation or receive a high router score. Reserve it based on measured results, not an assumption that every difficult-looking prompt needs it.

These are operational roles, not universal capability rankings. Evaluate the actual model versions and prompts you plan to run.

Metrics

  • Cost per request: Average cost across all tiers, weighted by request distribution.
  • Quality at a fixed evaluation set: Task-specific pass rate or rubric score, reported alongside cost so savings cannot hide degraded answers.
  • Escalation or strong-model rate: Percentage of requests that trigger another tier in a cascade, or are routed to the stronger model before generation.
  • Latency: P50 and P95 end-to-end latency, including verifier and escalation time.
  • Failure and fallback rate: Requests that exhaust the escalation budget, time out, or require a safe fallback.

Compare systems on the same held-out requests. Report the cost-quality curve across thresholds rather than presenting one favorable operating point as a general savings claim. Recalibrate when the workload or model pair changes.

Build This Pattern

Copy this prompt and paste it into Claude Code, OpenCode, Codex, or Cursor to implement this pattern.

Build a model cascade routing system for cost-optimized LLM usage. ROLE: You are a cost optimization system that routes requests to the cheapest capable model, escalating only when confidence or complexity requires it. CONSTRAINTS: - Three model tiers: fast-cheap (GPT-3.5), balanced (GPT-4), powerful-expensive (GPT-4-turbo) - Confidence threshold: below 0.7 escalates to next tier - Maximum 2 escalations per request; if all tiers fail, return error - Each tier has latency and cost budgets; respect per-tier limits - Track cost per request, latency per tier, and escalation rate for optimization TOOL CALLING: - Use function calling for: route_request(request_data, start_tier?), escalate_tier(current_tier, reason?), get_tier_stats(date_range?) - Each tool returns structured JSON with routing data and metadata STRUCTURED OUTPUT: - Routing decision must return JSON: { request_id: string, selected_tier: string, model: string, confidence?: number, escalated: boolean, escalation_reason?: string, estimated_cost: number } - Tier performance must return JSON: { tier: string, model: string, total_requests: number, avg_latency_ms: number, avg_cost: number, success_rate: number, escalation_rate: number } - Cascade stats must return JSON: { total_requests: number, by_tier: Record<string, number>, total_cost: number, avg_cost_per_request: number, escalation_rate: number, cost_savings_percent: number } CHAIN OF THOUGHT: - Initial routing: analyze request complexity → select cheapest tier → execute → evaluate confidence - Escalation: if confidence below threshold → move to next tier → re-execute → compare results - Cost tracking: log tier usage → calculate costs → identify optimization opportunities - Performance monitoring: track latency per tier → identify bottlenecks → optimize routing rules FEW-SHOT EXAMPLES: Request: 'What is 2+2?' Routing: { selected_tier: 'fast-cheap', model: 'gpt-3.5-turbo', confidence: 0.98, escalated: false, estimated_cost: 0.0001 } Request: 'Explain quantum computing in detail' Routing: { selected_tier: 'fast-cheap', model: 'gpt-3.5-turbo', confidence: 0.45, escalated: true, escalation_reason: 'low_confidence' } Escalation: { selected_tier: 'balanced', model: 'gpt-4', confidence: 0.89, escalated: false, estimated_cost: 0.003 } Stats: { total_requests: 1250, by_tier: { fast-cheap: 890, balanced: 310, powerful: 50 }, total_cost: 12.50, cost_savings_percent: 68 } EVALUATION CRITERIA: - Cost efficiency: percentage of requests handled by cheapest capable tier - Accuracy maintenance: answer quality maintained across tier escalations - Latency optimization: percentage of requests completed within tier latency budgets - Escalation accuracy: percentage of escalations that improve answer quality The system should: 1) Define model tier list: fast-cheap, balanced, powerful-expensive, 2) Route each request to cheapest tier first, 3) If confidence below threshold, escalate to next tier, 4) Handle all tiers failing, cascade timeout, 5) Handle requests needing specific model capabilities, 6) Handle models with different context limits, 7) Track cost per request, latency per tier, and escalation rate, 8) Verify correct tier selection based on request complexity.