Area 04 · Evals · Concept 10 of 20

Trajectory Evals

Judge the whole path of steps the agent took, not only its final answer.

01–10 · The basePhase 4 · Weeks 15–17≈ 1 hour

The idea

Two agent runs can reach the same answer, one in 4 sensible steps and one in 19 steps with repeated searches and a near-miss with a delete tool. Final-answer evals cannot see the difference. Trajectory evals look at the full sequence of actions.

Checks include: did it use required tools, did it avoid forbidden ones, did it follow the expected order, how many steps and tokens did it use, did it repeat itself? Some checks are code; some need a judge reading the trace with a rubric.

You need traces to do this, which is why tracing and evals go together.

Why it matters

Efficiency and safety live in the trajectory. That is where cost and risk come from.

Words to know

  • TrajectoryThe ordered list of model calls and tool calls in a run.

Try it without code

Review a trace

  1. Print the tool calls of one agent run as a numbered list.
  2. Mark any step you would remove.
  3. Write the ideal trajectory next to it.

Use it in code

Rule-based checks over a recorded trajectory.

python
def check_trajectory(calls, must_use=(), forbidden=(), max_steps=10):
    names = [c["name"] for c in calls]
    issues = []
    for t in must_use:
        if t not in names: issues.append(f"never used {t}")
    for t in forbidden:
        if t in names: issues.append(f"used forbidden {t}")
    if len(calls) > max_steps: issues.append(f"{len(calls)} steps > {max_steps}")
    repeats = {n for i, n in enumerate(names) if names[i:i+3] == [n, n, n]}
    if repeats: issues.append(f"3x repeated: {repeats}")
    return issues

print(check_trajectory([{"name": "search"}] * 3 + [{"name": "finish"}], must_use=["fetch_page"]))
PRACTICE · TICK IT AFTER THIS

Record trajectories for 10 research tasks and report average steps, repeated calls and rule violations.

Common mistakes

  • Requiring one exact path when several are fine.

Go deeper