Automate the investigation, not the fix

Teams leak faster than they bail, and unknown-return work always loses to features — “like taking something from the grocery store and going all the way to the cashier just to know how much it costs.” The wedge is a weekly agent surfacing scored, high-return opportunities so the deferred performance sprint simply happens; one payoff example was an N+1 query that had sat in the codebase for six years. Agents lack business context, not just runtime context, so humans still define what matters.

The same split appears in security: deterministic tools own detection — fixed cost, high confidence, reproducible — and models own reasoning, explanation and proposing fixes, because nondeterminism makes a model an unreliable oracle but an excellent explainer. The demonstration is unusually clean: eight of ten vulnerabilities found with the whole repo in context, and when context was narrowed to the single file containing a prototype-pollution bug and the same question asked, the finding disappeared — and swapping models did not recover it. “It’s not about models, it’s about the rest of things.”

Underneath both sits a prerequisite: an agent handed a rare or production-only failure with no runtime evidence either fails to reproduce it or guesses, and the guesses are systematically bad — blind query rewrites, or catching the exception as a “fix” for a systemic problem. Traces and a function-level map of real behavior — what each endpoint, consumer and cron actually invokes, with frequency, duration and failure status — convert speculation into diagnosis.

Also: CodeQL, Semgrep, ClickHouse, the Hud MCP server.

The detection/reasoning split is shipped in Assay, though not named that way: the evidence-row and closed-form-invariant layer is the deterministic tier, and the desk roles are the reasoning tier. The nondeterminism demonstration — the same bug found with the repo in context, missed with one file in context, and unrecoverable by swapping models — is a live demonstration of the Assay rule that an agent’s say-so is never evidence.

Scored opportunities as the output format. Impact × risk × effort, plus a one-line “why this is worth your attention”, generalizes the risk and gate annotations that briefs already carry to everything the desks surface to a human: triage output, findings, filed issues. Assay’s escalation vocabulary says what kind of attention is needed but not how much it is worth. Designed here, unbuilt. A false-positive tuning knob for the reviewer. Mine reviewer verdicts and worker dispute-replies, compute a per-rule reversal rate, and demote any rule reversed more than half the time to advisory — the curation ladder applied to the review role, on data that already exists in the pull-request record. Not built.

On runtime evidence, Assay’s position is narrower than the field’s. Assay’s memory surfaces distil lessons; the evidence here is the empirical case for preserving failure traces — the command, the output, the refusal — as first-class entries rather than only their distilled form, because the distilled form is exactly what a debugging agent cannot use. Not adopted. And the caveat that governs the whole concept: Assay has no production runtime-telemetry surface of its own to point an investigation agent at, so the investigation half of this concept is currently aspirational here rather than merely unbuilt.

• A comparison nobody has published: investigate-first versus fix-first on the same backlog. “The investigation is the leverage” currently rests on three consistent field reports and no controlled number.
• Whether the per-rule reversal-rate idea shows up anywhere as a shipped reviewer-tuning mechanism rather than a suggestion.