Outcome metrics over activity metrics

The list is long and consistent across independent orgs: lines of code, commits, pull requests opened, tokens consumed, sessions run, tool calls, initializations, weekly users, percent-AI-written, statement coverage, agent count. Each indicates that something is being used. None survives contact with the question of whether the work was worth doing — 90–100% AI-written code is compatible with essentially none of it written without human critical thinking, and 100% statement coverage is reachable by an agent writing trivial tests to close the gaps.

Merge rate, and specifically the share of merged work needing no human fix commit, as the adoption number — with a sub-100% rate as the healthy target, since 100% means rubber-stamping. Turns-to-correct: how many turns before the agent does the right thing, which measures the harness and its context rather than the model, and whose rising trend on one workflow is the earliest signal that its context has rotted. Interventions per run, counted and then engineered away one recurring class at a time. And the acceptance funnel — the ratio of generated candidates that survive human review and land — which in first-hand accounts runs in the low single to low double digits, nothing like a benchmark pass rate.

Composite beats single-number: velocity, quality and reliability read together, because over-rotating on any one of them is how the vanity metrics got in to begin with. And per-individual gains do not aggregate — roughly 30% individual developer efficiency is reported widely and fails to appear at workflow level. The gap is process re-engineering, not tooling; “make the process faster with AI” is the wrong ask, and “reimagine the process, then work backward” is the right one.

Also: the Stanford AI Spend Index; the “ghost engineers” paper; the SemiAnalysis study of ~2,400 companies.

Assay’s own recurring reports count activity — briefs, pull requests, sessions — which is exactly the family every source above names as vanity. Cycle time, shelf life, incident time, bug counts and cost per change is the recommended wholesale replacement, and Assay has none of them (how it runs).

One thing here is genuinely shipped, and it is the strongest claim Assay can make in this concept: a brief’s definition of done is not met until the verify row exists and, for artifact-producing streams, the artifact is observed live (lifecycle). That verified-not-merged bar is the analogue of “adopted”, and it is what stops the equivalent of the unused-feature flood: merged-but-unverified output. The 30%-that-vanishes result is outside validation of the second half — individual gains die without process re-engineering, which is the cleanest external argument for a process with gates over a prompt library (desk roles).

A zero-human-touch merge rate, computable today from bot commit identities against the merged set, and the best available answer to whether the fleet is good or merely loud — with the target deliberately below 100%. Turns-to-correct, whose closest existing signals are reviewer findings per PR, verification rejections, and iterations to first green check. Interventions per brief or per PR, which names each recurring intervention class so it can be engineered away. Evidence-citation rate, directly measurable from what the desks already write. An acceptance funnel per role — a 73-landed-of-313-produced ratio is the realistic anchor for any throughput claim, and it validates the human merge gate rather than indicting it. And per-brief token spend, to answer whether Assay is over- or under-spending per unit of merged work. Two standing rules follow: pin the vanity list above any future metric design, and prefer a composite of velocity, quality and reliability over any single-number dashboard.

• Whether zero-human-fix merge rate becomes a published cross-org number rather than one vendor’s — a second independent figure would make it a usable external benchmark.
• The first results of the whole-repository study: ~150,000 engineers over a year is the only work at a scale that could settle the maintainability question.
• Whether anyone reports turns-to-correct with a number attached; today it is named but unquantified.