Area 04 · Evals · Concept 09 of 20

Tool Call Accuracy

Did the agent pick the right tool, with the right arguments?

01–10 · The basePhase 4 · Weeks 15–17≈ 1 hour

The idea

For agents, a good final answer depends on good tool calls. Tool call accuracy checks each decision: was a tool needed, was the right one chosen, and were the arguments correct (valid schema, right values)?

Build test cases as input + expected tool name + expected arguments (or argument checks like "city must be Dublin"). Run the model for one step only and compare. This is fast, cheap and deterministic to score.

Track wrong tool, missing call, unnecessary call and bad arguments separately; each has a different fix (descriptions, examples, schema constraints).

Why it matters

Tool selection errors are the most common agent failure and the easiest to test in isolation.

Words to know

  • Single-step evalTesting only the next action, not the whole run.

Try it without code

Write tool tests

  1. List 10 user requests for your agent.
  2. For each, write which tool should be called and with what arguments, or "none".

Use it in code

Score the first tool call against expectations.

python
CASES = [
    {"input": "Weather in Cork tomorrow?", "tool": "get_weather", "args": {"city": "Cork"}},
    {"input": "Thanks!", "tool": None, "args": {}},
]

def first_call(client, tools, text):
    msg = client.messages.create(model="claude-sonnet-5-5", max_tokens=300, tools=tools,
                                 messages=[{"role": "user", "content": text}])
    call = next((b for b in msg.content if b.type == "tool_use"), None)
    return (call.name, call.input) if call else (None, {})

def score(client, tools):
    ok = 0
    for c in CASES:
        name, args = first_call(client, tools, c["input"])
        ok += name == c["tool"] and all(args.get(k) == v for k, v in c["args"].items())
    return ok / len(CASES)
PRACTICE · TICK IT AFTER THIS

Write 20 single-step cases for your research agent and fix tool descriptions until accuracy is above 90%.

Common mistakes

  • Only testing full runs, where tool errors are hidden by recovery.
  • Exact-match on free-text arguments like search queries; check key terms instead.

Go deeper