Comparing Models: Parameters, Benchmarks, and What Actually Predicts Agent Performance

Parameters: what is known, what is not, and why it matters less than you think

None of the closed frontier models — the Gemini Pro line, Claude Opus and Sonnet, GPT-5 — publish parameter counts. Estimates circulate, none are confirmed, and in interviews the right answer is "undisclosed, and here is why it would not settle the question anyway". Open-weight models do publish counts: Gemma 3 tops out at 27B dense; Llama 4 Maverick is ~400B total with 17B active; Qwen3 has a 235B-total/22B-active flagship; DeepSeek-V3/R1 are 671B total with 37B active; OpenAI’s gpt-oss ships at 120B (5.1B active) and 20B.

Those "total / active" pairs are the important idea. Mixture-of-experts (MoE) models route each token through a few of many expert sub-networks, so a 671B model computes with only 37B parameters per token. Total parameters predict memory footprint and knowledge capacity; active parameters predict per-token latency and cost. Neither predicts agentic quality well: post-training (RL on tool use, long-horizon tasks, instruction following) now moves capability more than scale does, which is why a 2026 mid-tier model beats a 2024 flagship on agent benchmarks at a fraction of the size.

• •Dense vs MoE: "70B dense" and "400B MoE / 17B active" can have similar per-token cost with very different knowledge breadth
• •Context window is a separate axis: Gemini 2.5 Pro at 1M tokens, Claude Sonnet 4.5 at 200k (1M in beta), GPT-5 at ~400k — and effective context (quality at long lengths) is smaller than the headline for every model
• •Reasoning effort / thinking budget is a runtime dial on all three providers that changes benchmark scores by tens of points — comparisons without it stated are meaningless
• •Ask "what was it post-trained to do?" before "how big is it?"

The benchmark zoo, decoded

Every launch post quotes a wall of numbers. Each benchmark measures one narrow thing, and most were saturated or contaminated within a year of release. Learn what each one tests so you can map it to your workload — or ignore it.

How leaderboards mislead

Contamination: benchmark questions leak into training data, so scores measure memorisation. Refreshing benchmarks (LiveBench, LiveCodeBench) and private held-out sets exist precisely because of this. Scaffold effects: on agentic benchmarks the harness — retries, tool design, planning prompts — swings results by 10–20 points, so a lab’s number with its own scaffold is not your number with yours. Compute disparity: "with parallel test-time compute" or "high reasoning effort" can mean 10× the cost of the comparison model; check the footnotes. Selection: launch posts show the benchmarks where the model wins.

And the deepest problem: benchmarks are averages over tasks that are not yours. A model that is best on SWE-bench Verified may be third on your TypeScript monorepo with your build system. Public benchmarks shortlist candidates; only your eval picks one.

The three line-ups, by tier

All three providers ship the same shape of catalogue: a flagship for hard reasoning and long agentic work, a mid-tier that is the production default, and a small tier for routing, classification, extraction and high-volume chat. Names and versions turn over every few months — the tiers do not. Learn the tier logic and check the provider’s model page for the current names before quoting one in an interview.

Cost and latency are part of "best"

A flagship that is 3 points better on a benchmark and 10× the price is not the best model for a router that runs a million times a day. Compare per-million-token prices for input, cached input and output; check batch discounts (typically 50%) for offline work; and measure time-to-first-token and tokens-per-second at your typical context length, because a model that is fast at 4k tokens can be slow at 200k.

The winning production pattern is almost always a mix: a small model routes and extracts, the mid-tier does the work, the flagship handles the escalations and the evals. Lesson 1’s routing pattern and lesson 3’s worker/orchestrator split exist to make this mix natural.

Build the eval that decides

Fifty real cases from your workload beat every public leaderboard. Collect them from logs (anonymised), write an expected outcome for each, and score with the strictest checker you can: exact match, a test suite, a schema check, and an LLM judge only where nothing deterministic exists. Run every candidate model at the same reasoning setting and the same harness, three times, and report pass^3 (passes all three) rather than the mean — agents need reliability, not a good day.

Then keep it running. A model version bump, a prompt tweak, or a new tool description can move the score, and the eval is how you notice before your users do.

import json
from anthropic import Anthropic

client = Anthropic()
CASES = [json.loads(l) for l in open("eval/cases.jsonl")]  # {"input":..., "expected":...}

def run_case(model: str, case: dict, effort: str) -> str:
    resp = client.messages.create(
        model=model,
        max_tokens=4000,
        system=open("prompts/system.md").read(),
        thinking={"type": "enabled",
                  "budget_tokens": {"low": 1024, "high": 16000}[effort]},
        messages=[{"role": "user", "content": case["input"]}],
    )
    return next(b.text for b in resp.content if b.type == "text")

def passes(output: str, case: dict) -> bool:
    return check(output, case["expected"])       # exact / schema / tests — deterministic

def pass_k(model: str, effort: str, k: int = 3) -> float:
    ok = 0
    for case in CASES:
        if all(passes(run_case(model, case, effort), case) for _ in range(k)):
            ok += 1
    return ok / len(CASES)

for m in ["claude-haiku-4-5", "claude-sonnet-4-5"]:
    print(m, "pass^3 =", pass_k(m, "high"))