Agent Evals: Trajectories, Tool Calls, and Regression Suites

Evals for a chat feature grade an answer. Evals for an agent must grade a trajectory: the sequence of reasoning, tool calls, tool results, and intermediate outputs that unfolds over many turns before the final answer appears. An agent can reach the right answer by luck after calling the wrong tools six times, and it can reach the wrong answer after a flawless trajectory because a tool returned stale data. If you only grade the end, you cannot tell those apart, and you will ship regressions you never saw.

"AI evals engineer" became a job title because this work is distinct, hard, and decisive. This module teaches the version of it you need to ship agents: the three layers of agent evaluation, how to build a task suite that reflects reality, how to grade trajectories mechanically and with judges, and how to turn the suite into the regression gate that makes every other module's fixes permanent.

Three Layers

Layer	Question	Signal source	When
Final answer	Did it complete the task?	Held-out task suite with graded outcomes	Every change, in CI
Trajectory	Did it get there sensibly? Right tools, right order, no loops, recovered from errors	Same suite, graded on the trace	Every change, in CI
Per-turn, in production	Are real users being served?	Corrections, retries, escalations, thumbs, abandonment	Continuously

Public benchmarks (τ-bench for tool-using agents in customer-service-style environments, SWE-bench for coding agents) tell you about models. They are useful for choosing one and useless for knowing whether your agent, with your tools and your harness, works. They also saturate and get gamed. Your suite is the only eval that measures your system.

Building the Task Suite

The suite is a set of tasks, each with a starting state, an input, and a specification of success. Sources, in order of value:

1. Production failures. Every incident, every user correction, every "why did it do that" becomes a task. This is the never-again loop from harness engineering, closed.
2. Representative successes. Sample real tasks that worked, so the suite catches regressions on the common path, not only the edge cases.
3. Adversarial cases. Injections, ambiguous requests, tools that return errors, tools that return empty results, tasks that should be refused.
4. Synthetic variations. Take a real task and vary the entities, the phrasing, the order of information. Cheap breadth once you have a template.

Aim for 30 tasks before the first release, 100 within a quarter, and keep growing from failures. Each task needs a reproducible starting state: seed data, mocked or sandboxed tools with deterministic responses, and a fixed model version for the baseline run.

pythonCopy
@dataclass
class AgentTask:
    id: str
    input: str
    setup: callable                 # seeds the sandbox / fixtures
    expected_tools: list[str]       # tools that must be called (order-sensitive prefix optional)
    forbidden_tools: list[str]      # tools that must NOT be called
    max_turns: int
    check_final: callable           # (final_answer, world_state) -> bool
    rubric: str | None = None       # for judge-graded free text

Grading the Trajectory

Most trajectory grading is mechanical, and mechanical grading is what you want: cheap, deterministic, and unarguable.

• Required tools called. The expected_tools appear in the trace.
• Forbidden tools not called. The read-only task never touched update_record.
• Order, where it matters. read_ticket before add_note. Grade as a subsequence, not an exact match, so the agent has latitude.
• Turn count within budget. Over budget is a loop.
• Error recovery. When a tool returned an error, did the next call change something (args, tool) or repeat verbatim? Verbatim repeats are loops in disguise.
• Argument validity. Every call passed schema validation on the first try. Retries after validation failures are a description or schema problem.
• World-state check. The sandbox ends in the expected state: the note exists, the file was not deleted.

pythonCopy
def grade_trajectory(task: AgentTask, trace: list[dict]) -> dict:
    calls = [e for e in trace if e["event"] == "executed"]
    names = [c["tool"] for c in calls]
    refused = [e for e in trace if e["event"] == "refused"]
    repeats = sum(1 for a, b in zip(calls, calls[1:])
                  if a["tool"] == b["tool"] and a["args"] == b["args"] and not a["ok"])
    return {
        "required_present": all(t in names for t in task.expected_tools),
        "forbidden_absent": not any(t in names for t in task.forbidden_tools),
        "ordered": is_subsequence(task.expected_tools, names),
        "within_budget": len(calls) <= task.max_turns,
        "no_verbatim_retries": repeats == 0,
        "no_invalid_args": len(refused) == 0,
    }

Report these as separate metrics, not one blended score. "Trajectory pass rate" hides which layer regressed; "no_verbatim_retries dropped from 98% to 84%" tells you a tool started failing and the agent stopped recovering.

Judges, and Validating Them

Free-text outputs (summaries, explanations, drafted replies) need a rubric and a grader. An LLM judge is the practical grader, and it is only trustworthy after you have validated it.

1. Write the rubric as specific, checkable criteria ("mentions the refund amount", "does not promise a delivery date"), not vibes ("is helpful").
2. Have two humans grade 100 outputs against the rubric. Measure their agreement; if they disagree often, the rubric is the problem.
3. Run the judge on the same 100. Measure agreement with the humans. Above roughly 85% agreement on a rubric humans agree on, use it. Below that, fix the rubric or the judge prompt and repeat.
4. Re-validate when you change the judge model.

Known judge biases: longer answers score higher; the first of two options scores higher; a model prefers its own outputs. Mitigate with pairwise comparisons in both orders, length-controlled rubrics, and a judge from a different model family than the agent.

From Suite to Regression Gate

The suite earns its keep in CI. On every change to the prompt, the tools, the harness, or the model version:

• Run the full suite against a fixed sandbox.
• Compare each metric to the baseline. Block the merge on any drop beyond a threshold you chose in advance (start at 2 points on final-answer and trajectory pass rates; tighten as the suite matures).
• Attach the trace diff for failed tasks to the PR. Reviewers should be able to see what changed in the agent's behavior, not only that a number moved.
• Record cost and latency per task alongside quality. A change that raises pass rate 1 point and cost 40% needs a conversation.

Because agents are stochastic, run each task several times (three to five) and grade on the pass rate across runs. τ-bench formalizes this as pass^k, the probability that all k attempts succeed, which is the number a user experiences: they do not get to retry silently.

Production Signal

The third layer is the one that catches what the suite could not imagine. Instrument:

• Corrections. The user restates or fixes the agent's output in the next turn.
• Escalations. Handoff to a human, or a retry with a bigger model.
• Abandonment. The user leaves mid-task.
• Explicit feedback. Thumbs, ratings, "this was wrong".
• Route and cost drift. Escalation rate and cost per task moving without a deploy means the inputs changed.

Every production failure that gets triaged becomes a suite task. That flow, production to suite to CI, is the whole system, and it is what separates teams whose agents get better every month from teams whose agents oscillate.

Common Mistakes and Bad Instincts

• Final-answer only. The trajectory is where the regressions are.
• One blended score. Separate metrics or you cannot localize.
• Unvalidated judges. A judge you have not checked against humans is a random number with a confident tone.
• Single runs. Stochastic systems need pass rates.
• Benchmarks as your eval. They measure models, not your system.
• Suite that never grows. If failures do not become tasks, the loop is open.

Where to Go Next

• ai-security-for-agents-prompt-injection-excessive-agency-and-sandboxing: the adversarial tasks your suite needs
• observability-drift-feedback-loops-and-llm-evals: the production-signal layer in depth
• harness-engineering-the-runtime-around-the-model: the traces this module grades

What to Practice Next

For the agent you built in the harness and MCP modules, write a 30-task suite: 10 from failures you observed, 10 representative successes, 10 adversarial. Implement mechanical trajectory grading with the six checks above and a validated judge for any free-text output. Run every task five times, report pass rates per metric, and wire the suite into CI so a prompt change that drops any metric by more than 2 points fails the build. Then make one deliberate regression (remove a tool description) and show the gate catching it with the trace diff.