The idea
For agents, a good final answer depends on good tool calls. Tool call accuracy checks each decision: was a tool needed, was the right one chosen, and were the arguments correct (valid schema, right values)?
Build test cases as input + expected tool name + expected arguments (or argument checks like "city must be Dublin"). Run the model for one step only and compare. This is fast, cheap and deterministic to score.
Track wrong tool, missing call, unnecessary call and bad arguments separately; each has a different fix (descriptions, examples, schema constraints).
Tool selection errors are the most common agent failure and the easiest to test in isolation.
Words to know
- Single-step evalTesting only the next action, not the whole run.
Try it without code
Write tool tests
- List 10 user requests for your agent.
- For each, write which tool should be called and with what arguments, or "none".
Use it in code
Score the first tool call against expectations.
CASES = [
{"input": "Weather in Cork tomorrow?", "tool": "get_weather", "args": {"city": "Cork"}},
{"input": "Thanks!", "tool": None, "args": {}},
]
def first_call(client, tools, text):
msg = client.messages.create(model="claude-sonnet-5-5", max_tokens=300, tools=tools,
messages=[{"role": "user", "content": text}])
call = next((b for b in msg.content if b.type == "tool_use"), None)
return (call.name, call.input) if call else (None, {})
def score(client, tools):
ok = 0
for c in CASES:
name, args = first_call(client, tools, c["input"])
ok += name == c["tool"] and all(args.get(k) == v for k, v in c["args"].items())
return ok / len(CASES)Write 20 single-step cases for your research agent and fix tool descriptions until accuracy is above 90%.
Common mistakes
- Only testing full runs, where tool errors are hidden by recovery.
- Exact-match on free-text arguments like search queries; check key terms instead.