The idea
Most traffic is easy: greetings, simple lookups, short rewrites. Sending all of it to your most capable model wastes money and time. A router decides per request which model should handle it.
Routers range from simple rules (by feature, input length or keywords) to a small classifier model that predicts difficulty, to cascades (try the cheap model, check confidence or validate output, escalate if it fails).
Measure the router itself: how often it sends hard requests to the cheap model (quality loss) and easy ones to the expensive model (wasted cost). Tune the threshold with your eval set.
Routing is often the largest single cost reduction in a mature AI product.
Picture it
Words to know
- RouterLogic that picks a model per request.
- CascadeTry cheap first, escalate on failure.
Try it without code
Label your traffic
- Take 30 real or sample requests.
- Mark each easy or hard.
- Estimate the saving if easy ones went to a model 5× cheaper.
Use it in code
A classifier router with a rule shortcut.
import anthropic
client = anthropic.Anthropic()
SMALL, LARGE = "claude-haiku-5-5", "claude-sonnet-5-5"
def route(text):
if len(text) < 80 and "?" not in text:
return SMALL # cheap rule for chit-chat
verdict = client.messages.create(model=SMALL, max_tokens=3, temperature=0,
messages=[{"role": "user", "content":
f"Is this request simple (lookup, rewrite, short answer) or complex (multi-step reasoning, code, analysis)? Reply SIMPLE or COMPLEX.\n{text}"}]
).content[0].text.strip()
return LARGE if verdict.startswith("COMPLEX") else SMALL
print(route("thanks!"), route("Design a schema for multi-tenant invoices with audit history."))Route your golden set, compare quality and cost against always-large, and report the trade-off.
Common mistakes
- A router that costs nearly as much as it saves.
- Never re-evaluating routing after model upgrades.