Model cascade routing reduces cost and latency by sending each request to the cheapest capable model first. Only when the initial model cannot produce a satisfactory result is the request escalated to a more powerful (and expensive) model.
How it works
Tier system -- models are organized into tiers by capability and cost.
Confidence check -- after a model produces an output, a confidence estimator evaluates whether the result is acceptable.
Escalation -- if confidence is below threshold, the request is forwarded to the next tier.
Tiers
Fast-cheap: Small models for simple tasks (classification, extraction, formatting).
Balanced: Mid-size models for most routine reasoning and generation tasks.
Powerful: Large frontier models reserved for complex reasoning, creative work, and edge cases.
Metrics
Cost per request: Average cost across all tiers, weighted by request distribution.
Escalation rate: Percentage of requests that require a higher tier.
Latency per tier: P50 and P95 response times for each model tier.
Build This Pattern
Copy this prompt and paste it into Claude Code, OpenCode, Codex, or Cursor to implement this pattern.
Build a model cascade routing system for cost-optimized LLM usage.
ROLE: You are a cost optimization system that routes requests to the cheapest capable model, escalating only when confidence or complexity requires it.
CONSTRAINTS:
- Three model tiers: fast-cheap (GPT-3.5), balanced (GPT-4), powerful-expensive (GPT-4-turbo)
- Confidence threshold: below 0.7 escalates to next tier
- Maximum 2 escalations per request; if all tiers fail, return error
- Each tier has latency and cost budgets; respect per-tier limits
- Track cost per request, latency per tier, and escalation rate for optimization
TOOL CALLING:
- Use function calling for: route_request(request_data, start_tier?), escalate_tier(current_tier, reason?), get_tier_stats(date_range?)
- Each tool returns structured JSON with routing data and metadata
STRUCTURED OUTPUT:
- Routing decision must return JSON: { request_id: string, selected_tier: string, model: string, confidence?: number, escalated: boolean, escalation_reason?: string, estimated_cost: number }
- Tier performance must return JSON: { tier: string, model: string, total_requests: number, avg_latency_ms: number, avg_cost: number, success_rate: number, escalation_rate: number }
- Cascade stats must return JSON: { total_requests: number, by_tier: Record<string, number>, total_cost: number, avg_cost_per_request: number, escalation_rate: number, cost_savings_percent: number }
CHAIN OF THOUGHT:
- Initial routing: analyze request complexity → select cheapest tier → execute → evaluate confidence
- Escalation: if confidence below threshold → move to next tier → re-execute → compare results
- Cost tracking: log tier usage → calculate costs → identify optimization opportunities
- Performance monitoring: track latency per tier → identify bottlenecks → optimize routing rules
FEW-SHOT EXAMPLES:
Request: 'What is 2+2?'
Routing: { selected_tier: 'fast-cheap', model: 'gpt-3.5-turbo', confidence: 0.98, escalated: false, estimated_cost: 0.0001 }
Request: 'Explain quantum computing in detail'
Routing: { selected_tier: 'fast-cheap', model: 'gpt-3.5-turbo', confidence: 0.45, escalated: true, escalation_reason: 'low_confidence' }
Escalation: { selected_tier: 'balanced', model: 'gpt-4', confidence: 0.89, escalated: false, estimated_cost: 0.003 }
Stats: { total_requests: 1250, by_tier: { fast-cheap: 890, balanced: 310, powerful: 50 }, total_cost: 12.50, cost_savings_percent: 68 }
EVALUATION CRITERIA:
- Cost efficiency: percentage of requests handled by cheapest capable tier
- Accuracy maintenance: answer quality maintained across tier escalations
- Latency optimization: percentage of requests completed within tier latency budgets
- Escalation accuracy: percentage of escalations that improve answer quality
The system should: 1) Define model tier list: fast-cheap, balanced, powerful-expensive, 2) Route each request to cheapest tier first, 3) If confidence below threshold, escalate to next tier, 4) Handle all tiers failing, cascade timeout, 5) Handle requests needing specific model capabilities, 6) Handle models with different context limits, 7) Track cost per request, latency per tier, and escalation rate, 8) Verify correct tier selection based on request complexity.