Production outcome evals

Always-on outcome evals: rubric-comparing judge agents scoring real outcomes post-merge, with no engineer in the loop. Shadow mode: agent and human work the same live queue in parallel, and every disagreement is both the measurement and the training signal. Causal in-the-wild evaluation: randomize model, orchestrator or tool assignment across real user sessions, so a leaderboard position is a treatment effect with a confidence interval rather than a static score waiting to be overfit.

The layer that ties these together runs from business outcome down to diagnostics — L0 outcomes, L1 black-box task success, L2 component evals, L3 diagnostics. Its named anti-pattern is hill-climbing L2 and L3 while L1 stays flat, which is what happens when the levels are not causally linked. L0 is resolution rate, cost per ticket, analyst hours saved; L1 is whether the refund was the correct amount.

Run as a standing engine rather than a gate, the whole thing is an improvement loop: traces in, failures auto-categorized, updates proposed to both the agent and the eval suite, humans accept or reject. The reporting question shifts with it — not “is your eval above 90%” but what did you change because of it, and how stale is the dataset that produced the number.

Also: Azure SRE agent fleet; Arena.

Assay’s verify rows run before a merge and run once. That is the honest position: every mechanism above is an evolution beyond where Assay sits today. What is shipped is the piece Microsoft names — the agent validates its own outcome and presents the evidence before a human reviews, rather than the human reading the diff — and the claim-plus-evidence release gate is the same instinct applied to merges (lifecycle). What is not built is anything always-on: no post-merge standing evaluation, no judge agent scoring live outcomes, and no re-run of a brief’s verify rows after the world moves under them.

Assay’s verify rows live almost entirely at L1 and L2 — did this brief’s deliverable work — with nothing tying a landed brief back to whether a higher-level outcome improved. That absence is precisely why the recurring reports count activity. The concrete steal is an L0 line per brief naming the outcome metric that should move; it would be a schema change to the brief format, and it has not been proposed.

The eval-design-review-before-build gate is the same shape as the review gate a brief passes before it is authored — arrived at independently, which is worth recording as corroboration rather than as a gap. Randomized causal evaluation needs traffic volume a single-team methodology does not have, but the discipline of defining the estimand before counting does port, and it is the standard any future “measure desk quality from outcomes” work should be held to. Shadow mode — agent proposes, human does, the diff feeds learning — is a complementary loop to the unbuilt transcript-mining pass, and Uber’s traces → categorized failures → proposed updates → human accept/reject is effectively that pass in production, with useful evidence that the accept/reject gate is the right terminus rather than a compromise.

• Whether always-on post-merge evaluation shows up outside large platform teams — Uber, Microsoft and Ironclad all run dedicated agent-platform orgs, and a small-team version would make it adoptable.
• Published dataset-staleness practice: “a five-month-old dataset is a liability” is the only number the field has offered on eval decay.
• Any account of an L0 metric actually attributed to agent work — the stack is well described, the attribution is not.