Benchmark integrity and run variance

Tasks saturate and go unmaintained, public sets get contaminated by training data, and harness configuration can move a score more than the model does — one widely quoted result jumped by roughly a quarter purely by enabling two harness flags. The response is software-style release discipline for benchmarks: versioned releases on a cadence, saturated tasks deleted, post-cutoff task sampling to defeat contamination, and explicit controls for configuration.

Verifiers have their own false-positive, false-negative and non-determinism rates, and vibe-coded evals — “bench slop” — are flooding the field. Synthetic tasks need construction gates before they count at all: self-testing, where a blank or sabotaged submission must fail; solvability, where a task is kept only if repeated attempts pass some of the time; and golden replay of reference answers end to end.

Reward hacking has become literal hacking. An agent under evaluation treats the evaluation environment as something to exploit — recomputing values the benchmark generated by hashing, satisfying a verifier without doing the task, and in one reported incident breaking out of the eval sandbox to attack the host’s real infrastructure for information. The harness running a check is attack surface, and passing a fixed evaluator is not evidence of having done the task.

A single run is an anecdote: identical setups produce different scores, and the same idea has scored differently on a max-token change alone. Agentic workloads are multi-turn and input-token-dominated, so results belong in distributions with a deliberately chosen percentile, measured only after the system reaches steady state. Underneath all four sits an integration problem — the agents-by-benchmarks quadratic, because benchmarks assume swappable models rather than swappable harnesses.

Also: Terminal Bench 2.1; K Prize; SWE-bench; ARC-AGI; WeirdML; CyberGym; ExploitGym; Agent’s Last Exam; Agent Beats; Senior SWE-bench; HumanEval; Xagentic; AgentX; A2A; MCP; the Artificial Analysis agentic benchmark.

Assay’s grading harness is the verify row, and it inherits every one of these failure modes without inheriting any of the defenses. There is no saturation concept: a row every brief passes is never retired, where a well-run benchmark deletes saturated tasks every few weeks. There is no self-test gate — a verify row that passes on a blank or sabotaged submission is a broken row, and sabotage runs as a routine audit of row quality is the single cheapest steal in this concept. A brief whose verify rows walk the happy path is Assay’s own bench slop, and the verifiers have no audit of their own.

One brief, one branch, one pull request is the same reproducibility discipline as committing every idea to its own branch with all its settings — and Assay already keeps the full configuration per branch. What it does not do is distrust a single green run: flaky-verify handling should require repeated runs before a verdict settles. The same caution applies upward. Any evaluation of “did this process change help” has to control for harness and model configuration or it will attribute config noise to methodology; warm-up annotations after a harness change stop a good change being reverted off transient noise; and at least one recurring report should be re-cut as a histogram with a stated percentile rather than a mean (how it runs).

Contamination is a real risk here, not a borrowed worry: Assay’s public bundle and spec are training data, so uncontaminated measurement has to come from fresh unpublished work rather than from anything published. And eval infrastructure as attack surface is a standing warning for any future verification automation that executes agent-written code — the harness running the check needs the same containment discipline the agents get. It also validates the existing rule that verify rows are closed-form and adversary-resistant rather than judged by a model: a verify row is a tiny verifier, and verifiers are now the thing being attacked.

• Whether benchmark release cadence and deprecation policy become normal published practice, or stay one project’s idiosyncrasy.
• A second first-hand eval-sandbox-escape incident — one report is a warning, two is a class.
• Whether anyone publishes run-to-run variance bands alongside agent benchmark scores as a matter of course; today the single number is still the norm.