Trajectory grading

The questions a trajectory answers are not the ones a final answer answers. Was each hop correct? Was the trajectory adequate to the stated intent? What did it cost in tokens and latency? Did it dead-end and backtrack, did it recover from its errors — and was the method it used one the task actually permitted. That last one is the sharpest: an agent asked to detect a condition from a measurement trend can instead read the diagnosis already written in the notes, and score full marks on a grader that only sees the answer.

The failure information an outcome eval discards turns out to be most of it. Correctness is one of four buckets — correctness, completeness, usefulness, efficiency — and the other three never appear in a final-answer score. Efficiency is not merely wasteful either: an inefficient trajectory causes downstream errors by exhausting the context it needed later. Trajectory metrics such as first-pass success, dead-end rate and unique-sequence variance get promoted to co-equal with answer quality.

Trajectories also carry free signals that predict failure before the outcome does — the agent’s own text degrading under stress reads as a health indicator straight off the surface. And at the top of the ladder sits a third evaluation tier above benchmark hill-climbing and meta-benchmarks: grading how an improvement was achieved, scored for no cheating, creativity and elegance.

Also: M-TRACK; self-RAG; adaptive RAG; Senior SWE-bench; Arena.

Assay holds everything trajectory grading needs and throws almost all of it away. Session transcripts, tool-call logs, reviewer findings and review disputes are all persisted; the reporting layer turns them into activity numbers — briefs, PRs, sessions. Nothing inspects a trajectory. Verify rows are per-step falsifiable checks on outcomes: they run after the fact and say nothing about the route taken to produce them.

Shipped: the verify role reads sessions rather than only re-running checks, which is the manual, unmeasured version of trajectory grading; reviewer findings per PR is the closest existing proxy for a dead-end rate (desk roles). Designed but unbuilt: the transcript-mining pass over session history, from which intervention rate, dead-end and backtracking rates and recovery rate are direct steals. Not designed: a trace-level metric of any kind. “Tokens per verified brief” and “wasted hops per brief” are the concrete upgrades over activity counting, and neither exists.

The forbidden-method case is a direct upgrade instruction for the brief schema: some verify rows must specify how a result may be obtained, and a reviewer should treat a method violation as a failure even when the output looks right. That is a spec change rather than a tooling change, and it has not been proposed.

• Whether anyone publishes a trajectory-metric set stable enough to compare across harnesses — dead-end rate, first-pass success and unique-sequence variance are each defined locally by one team.
• Whether cheap lexical health signals such as the “not-okay ratio” generalize beyond training-time divergence detection into deployment-time monitoring.
• A worked example of method-constrained grading in a software task rather than a regulated-domain one, which would make the verify-row spec change concrete.