Area 05 · AI System Design · Concept 01 of 20

Model Routing

Easy requests go to a cheap, fast model; hard ones go to a strong model.

01–10 · The basePhase 6 · Weeks 21–23≈ 1 hour

The idea

Most traffic is easy: greetings, simple lookups, short rewrites. Sending all of it to your most capable model wastes money and time. A router decides per request which model should handle it.

Routers range from simple rules (by feature, input length or keywords) to a small classifier model that predicts difficulty, to cascades (try the cheap model, check confidence or validate output, escalate if it fails).

Measure the router itself: how often it sends hard requests to the cheap model (quality loss) and easy ones to the expensive model (wasted cost). Tune the threshold with your eval set.

Why it matters

Routing is often the largest single cost reduction in a mature AI product.

Picture it

Words to know

  • RouterLogic that picks a model per request.
  • CascadeTry cheap first, escalate on failure.

Try it without code

Label your traffic

  1. Take 30 real or sample requests.
  2. Mark each easy or hard.
  3. Estimate the saving if easy ones went to a model 5× cheaper.

Use it in code

A classifier router with a rule shortcut.

python
import anthropic
client = anthropic.Anthropic()
SMALL, LARGE = "claude-haiku-5-5", "claude-sonnet-5-5"

def route(text):
    if len(text) < 80 and "?" not in text:
        return SMALL                                   # cheap rule for chit-chat
    verdict = client.messages.create(model=SMALL, max_tokens=3, temperature=0,
        messages=[{"role": "user", "content":
            f"Is this request simple (lookup, rewrite, short answer) or complex (multi-step reasoning, code, analysis)? Reply SIMPLE or COMPLEX.\n{text}"}]
    ).content[0].text.strip()
    return LARGE if verdict.startswith("COMPLEX") else SMALL

print(route("thanks!"), route("Design a schema for multi-tenant invoices with audit history."))
PRACTICE · TICK IT AFTER THIS

Route your golden set, compare quality and cost against always-large, and report the trade-off.

Common mistakes

  • A router that costs nearly as much as it saves.
  • Never re-evaluating routing after model upgrades.

Go deeper