You Cannot Evaluate an Agent by Its Answer

Trajectory-level evaluation and why turn-level scoring hides the failures that matter

Sep 30, 2026

A retailer’s refund agent completes a ticket, and the final message informs the customer that $84 has been returned to the original card. The order status updates to refunded, the test harness verifies the result against the reference and records a pass, and the team’s dashboard displays 100 percent.

In the same run, the agent might skip the required identity check, access two other customers’ records while searching for the order, and call one tool 27 times after a timeout. The closing message remains identical to that of a clean run. The scoring system cannot distinguish between the two runs.

This gap was less significant when models generated text only for human review because the human served as the check. Agents act directly, and evidence of their actions exists only in the run record, not in the final message.

Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls as reasons. Answer scores provide no insight into cost or risk control. Teams that grade only answers cannot address these concerns in their reports.

The unit to score is the entire run. The sections below explain how to do that, with the arithmetic and a worked example.

What an answer leaves out

A chat model generates text for a person to read. Grading the text evaluates what the user will see, and the person often catches errors the grader misses. An agent, however, performs actions. Its closing message only summarizes what the agent claims to have done.

Anthropic’s engineering team separates the transcript, which holds every message, tool call and result in a run, from the outcome, which is the state of the environment when the run ends. Their example is a flight-booking agent that tells the user the flight is booked, even though the reservation database contains no record of the reservation. Sierra built τ-bench, a benchmark for customer service agents, on the same premise. It scores a run by comparing the database at the end of the conversation with the intended end state.

Even this check is limited, as the τ-bench authors acknowledge. For example, an agent may issue a return without the required customer confirmation, yet still leave the system in the correct state.

Actions carry consequences that a closing message never shows. In July 2025, Jason Lemkin, founder of the SaaS community SaaStr, reported that Replit’s coding agent deleted his production database during a code freeze, wiping records on more than 1,200 executives and more than 1,190 companies. Replit’s CEO called the deletion unacceptable, saying it should never have been possible, and the company announced the automatic separation of development and production databases. Every command the agent ran is recorded in that session. A grader that reads only the final message never examines any of them.

The reverse happens too. Anthropic reported that Claude Opus 4.5 found a loophole in an airline booking policy on a τ2-bench task and produced a better result than the task anticipated. The benchmark scored the run as a failure because it did not match the task as written. A check that reads only the result penalized the better solution.

A chat model can fail only in the message it returns. An agent can fail at any step, and most steps are not visible to the user. Errors such as retrieving the wrong record, using an incorrect argument, or skipping a required policy document do not appear in the transcript unless someone reviews it. The final message is a self-written summary, and agents are trained to produce plausible closing statements even after flawed runs.

Reliability comes from repeated runs

An agent does not follow the same path twice on the same task. The τ-bench paper introduced pass^k, the chance that an agent succeeds on all k runs of a task, averaged across tasks. In that paper, GPT-4o agents solved fewer than half the tasks, and in the retail domain the chance of eight successes in a row fell below 25%.

A 90% per-run success rate may appear sufficient. However, requiring five consecutive successes reduces this to 59%, and ten in a row drops it to 35%. These figures assume uniform task difficulty and independent runs. In practice, tasks vary in difficulty, so calculations are performed per task and then averaged. For example, a task solved 90% of the time passes ten consecutive runs only 35% of the time, while a task solved half the time passes ten in a row less than once in a thousand attempts.

For each task, take n runs with c successes. The τ-bench paper’s unbiased estimate calculates the number of ways to choose k runs from the c successes, divided by the number of ways to choose k runs from all n. This calculation is concise in Python code.

from math import comb

def pass_hat_k(results, k):
    """results maps task id to a list of booleans, one per run, with n >= k."""
    scores = [comb(sum(runs), k) / comb(len(runs), k) for runs in results.values()]
    return sum(scores) / len(scores)

Select k based on how often the same situation occurs. For example, if a request type appears ten times daily, the probability of no failures in a day for that type is pass^10.

Public benchmarks offer limited support in this area. A 2026 preprint on long-horizon agent reliability notes that SWE-bench and WebArena report only a single attempt per task and do not analyze variance. Leaderboard rankings do not reflect consistency.

Three levels of scoring

A run represents a single attempt at a task. Its record, known as a trace or trajectory, lists every message, tool call, argument, and result in sequence. Scoring can be applied to individual steps, specific checkpoints, or the entire run.

Most teams already run the turn level, since it extends what they built for chat. Two metrics apply to each step. Tool selection assesses whether the agent chose the right tool and passed valid arguments, while step-level faithfulness assesses whether its statements about the result match what the tool returned. Google’s Vertex AI evaluation service scores tool selection with precision and recall against a reference set of calls. This includes precision (the proportion of the agent's calls that were relevant) and recall (the proportion of required calls it made). Turn checks are cheap and run on every step of every run. They also pass every step of the failing run in the example below because each step, taken alone, is well-formed.

A milestone is a checkpoint that the task must reach, checked against the trace or the system state. Plan quality is assessed at this level by determining whether the run achieved the necessary checkpoints. Most milestones are binary and can be checked programmatically, such as verifying identity or recording a refund with the correct method. Sequence matters only when policy specifies an order, such as ensuring verification occurs before a refund by comparing timestamps.

Trajectory scoring evaluates the run as a whole. It enforces limits on steps and spending, checks which records the agent accessed, and identifies loops. It also tests error recovery, assessing how the agent responds to timeouts or empty results. For example, an agent who repeats the same call 27 times handles a timeout differently from one who consults the customer or escalates the case. Trajectory scoring also measures consistency across repeated runs.

Cost should be measured at the trajectory level because it varies across runs. The difference between a clean run and a problematic one is critical for financial oversight. Anthropic found that agents use about four times the tokens of a single chat interaction, and a full multi-agent system uses about fifteen times as many. Their conclusion was that such architectures are justified only for tasks whose value offsets the increased cost. Including cost in trajectory scoring allows programs to evaluate this on a task-by-task basis, rather than relying on assumptions. Gartner cites escalating costs as a key reason for project cancellations, and the lack of trajectory-level cost data means teams may only discover this issue during financial reviews.

Grade the path against rules

Google’s Vertex AI evaluation service scores an agent’s tool calls against a reference sequence using six built-in metrics, including exact match, in-order match, any-order match, precision and recall. Anthropic’s engineers report that checking for a specific sequence of tool calls proved too rigid because agents find valid approaches the test’s authors did not anticipate, and they advise grading the agent's output based on the steps it took. The same engineers still check the path where policy calls for it. Their sample test for a support agent requires an identity check, caps the refund amount and limits the run to ten turns.

A three-part rule set settles the difference. It includes:

Required actions, with an order only where policy sets one
Prohibited actions
Budgets for steps and spend

Any approach that meets the rules passes, allowing the agent flexibility in its methods. Plan quality is evaluated separately from the rule set. It considers whether the agent’s initial actions were appropriate based on available information, not just whether the final outcome was correct. For example, an agent that issues a refund before checking the order history may arrive at the correct amount by chance. While a milestone check would pass this run, a plan check that reviews the sequence of tool calls would not, as gathering information before acting is essential for a reliable process. A run with the correct outcome but an improper sequence signals a potential issue.

A worked example

The following example is a composite built for this issue. The task, policy and numbers are illustrative.

A retailer’s agent handles refund requests. The test for a single task encodes the policy in a Markdown file that the code can read.

task: refund_damaged_item
input: "Order A-2291 arrived damaged. I would like my money back."
outcome:
  database:
    refunds: {order: A-2291, amount: 84.00, method: original_card}
required:
  - verify_identity succeeds before issue_refund
  - get_policy("refunds") called before issue_refund
prohibited:
  - read of any customer record other than the requester's
  - issue_refund above 100.00 without an approval_id
budget:
  tool_calls: 10
  cost_usd: 0.40
runs: 5

Run 3 of the five produced this trace.

Each step is well-formed, and the final message matches the tool result. All six turn checks pass. However, the identity check was still pending at step 4, and the agent issued the refund without customer confirmation. Additionally, the name search in step 2 returned records for three customers, leading the agent to access two accounts that were not the requester’s.

The five runs together look like this.

The outcome rate is 5/5. The rate on the full rule set is 2/5, and the task fails pass^5.

Each scoring level identified different issues. Turn checks found no problems, as every call was well-formed and each message matched its tool result. The milestone check detected skipped identity verification in runs 3 and 5 because the refund was issued before verification was completed. The access of out-of-scope records in run 3 and the 31 calls in run 4 were identified only at the trajectory level, where a timeout triggered a costly retry loop. All findings were detected by code reviewing the trace and database, without a judge model.

The identity failures point to the tool. The prompt instructed the agent to verify identity, and the agent complied in 3 of 5 runs. A refund tool that rejects any call from an unverified session complies in five of five. Evaluation showed how much the written instruction was worth as a control, and the tool supplies the control that holds.

A judge model is appropriate for evaluating whether the closing message explains the refund clearly and uses the correct tone. This requires subjective judgment, which code cannot provide.

Where a judge model holds up

A judge model is a second model that grades the first. The evidence on judges splits by task.

Judges do well with a verdict on an entire run against written criteria. An iOSWorld study of phone agents compared a trajectory-level judge with four human annotators on 128 trajectories and found 89% agreement on task success, with a Cohen’s kappa of 0.77. The LiveMCP-101 benchmark, which tests agents that call tools through the Model Context Protocol, reported kappa above 0.78 between human and judge ratings of trajectories on 30 sampled tasks across six models. Both are recent preprints with small samples, and neither tested judges on specific domains.

Judges do poorly at finding the step where a run went wrong. The Who&When benchmark, published at ICML 2025, gathers failure logs from 127 multi-agent systems. Its best method named the responsible agent 53% of the time and the decisive step 14% of the time. The TRAIL benchmark from Patronus AI holds 148 annotated traces with 841 errors, and the best model scored 11%. Both benchmarks were run on 2025 models. A 2026 preprint on a method built for the job reports 46% step accuracy on the algorithm-generated half of Who&When and 29% on the hand-crafted half. A step accuracy under half cannot support an incident report. Patronus notes that errors compound at every step and through the systems an agent touches, and that its traces can exceed the context window of the model that reads them.

The key is to use each tool for its strengths. The judge model provides a verdict on written criteria for the entire run. Code identifies specific steps, as a structured trace allows queries on tool names, arguments, and timestamps.

Anthropic’s guidance for judges carries over to trajectories. It includes:

Write a structured rubric.
Grade each dimension with its own judge.
Let the judge answer “unknown” when the trace lacks the evidence.
Calibrate against human graders.

It’s critical to test the judge before trusting it. A 2026 preprint on judge reliability, BabelJudge, names two failure modes to probe in trajectory judges, a preference for longer traces and blindness to wrong tool arguments. Both can be tested directly. Copy a passing trace and pad it with redundant steps, and the score should hold or fall. Copy it again, change one argument to an incorrect value, and the score should drop. A judge that fails either test has no place in a release decision.

Calibration does not require a large sample. Fifty traces, independently graded by two people using the same rubric as the judge, are sufficient to measure agreement and identify discrepancies. When disagreements arise, review the trace to determine the correct outcome, then adjust the rubric or narrow the judge’s scope as needed. Because a judge calibrated only once at launch will drift over time as production data changes, schedule regular recalibrations rather than waiting for issues to appear.

Who sets the floor

Trajectory evaluation generates quantitative results, but someone must set the thresholds these results must meet before deployment. This responsibility should not fall to the engineering team that developed the agent.

The system owner, who is accountable for its ongoing performance and compliance, sets the minimum acceptable thresholds. This person is typically part of the business unit affected by the agent’s actions, not a central AI function. The thresholds include the pass^k rate, aligned with task frequency, and zero tolerance for prohibited actions involving money, health, legal exposure, or other customers’ data.

Thresholds should reflect the level of risk. A drafting agent for internal memos can operate with a modest pass^k threshold, as errors only require rewrites. A refund agent handling financial transactions requires a high outcome rate and zero tolerance for prohibited actions, since mistakes can result in financial loss or erode customer trust. Applying a single standard to all agents ignores the differing risks between, for example, a memo drafter and a payment processor.

Trajectory evaluation is essential for production approval, not just for internal engineering use. The evaluation suite provides the evidence required by reviewers. The system owner reviews the results and determines if the agent meets the necessary standards for its intended use. These roles are distinct. Engineers should not set their own thresholds, and reviewers should not approve agents without evaluation data.

What to report

Four numbers describe an agent to leadership, and a program should produce them on demand.

Outcome rate. The share of runs where the system state matches the goal, from database checks and never from the agent’s own words.
Rule-clean rate. The share of runs with every required milestone met, no prohibited action and the budget kept, reported as pass^k at the k that matches how often the situation recurs.
Cost per successful run at the median and 95th percentile.
Recovery rate. The share of runs with a deliberately injected fault, such as a timeout, an empty result or two records that disagree, that end correctly or hand the case to a person.

The release rule is based on these metrics. Set zero tolerance for prohibited actions, and run a sufficient number of tests to ensure statistical significance. Zero violations in n runs supports a maximum true rate of approximately 3 divided by n at 95% confidence. For example, 200 clean runs set a ceiling of 1.5%. However, an agent processing 10,000 requests daily at this rate could still produce 150 violations per day, so the test suite size must reflect expected volume. For other checks, establish a pass^k threshold for each risk tier and a cost ceiling at the 95th percentile.

Gartner’s forecast of cancellations gives cost and risk control as two of its causes. These four numbers are the evidence leadership would ask for on both, and the outcome rate covers the third cause, business value, by tying results to system state.

A four-phase rollout

Weeks 1 to 3: Document the rules and enable full trace capture for every agent action, including arguments and results, before starting any scoring. A rule is only effective if there is trace data available for evaluation. For each agent, compile a list of prohibited actions based on policy, legal, and security requirements, and assign ownership to someone other than the agent’s developer. Gather 20 to 50 tasks from real failures and support tickets. According to Anthropic’s engineers, this sample size is sufficient for initial evaluation, as early changes have significant impact.

Weeks 4 to 8: Use code to verify system state and milestones. Implement database checks for outcomes and code checks for milestones and prohibited actions using trace data. Run each task five times. Report both the outcome rate and the rule-clean rate together, as the difference between them is often a key finding and may be the first time it is quantified for stakeholders.

Weeks 9 to 12: Introduce judge models where code-based checks are insufficient, such as for tone, explanation quality, and escalation message content. Assign a separate judge to each criterion and allow “unknown” as a possible response. Have two people independently grade 50 traces to compare with the judge’s assessments. Conduct padding and argument tests before including judge scores in reports.

Week 13+: Begin production scoring. Sample live traces daily and apply the same code-based checks. Immediately alert on any prohibited action, as issues like payments or data exposures require prompt attention. Convert each failure into a new task. Move tasks that consistently pass into a regression suite, which should be run after any changes to prompts, tools, or models. This approach ensures new models are fairly evaluated before replacing existing ones, which is increasingly important as model updates become more frequent.

Each phase yields a metric that leaders can understand without technical detail. In Phase 2, the gap between outcome rate and rule-clean rate is often the key factor influencing funding decisions.

The risk that remains

The primary risk lies in the approval process. Leadership may expand an agent’s authority based on consistently high scores, even though these scores are derived from metrics that cannot reveal violations. The issue is not misinterpretation, but reliance on incomplete evidence.

Trusted AI

Discussion about this post

Ready for more?