Control taxonomy and calibrated LLM judges

The design question for any given rule is which quadrant owns it and where on the path to production it sits: in-session when cheap, then pre-commit, then the pull request, then CI, then a weekly cadence. Guides subdivide further into normative, informative and instructional — and include computational guides, which is the underrated quadrant: the headline “hundreds of developer-years saved” on one large Java upgrade was mostly codemods made available to the model, not the model reasoning harder.

Within the inferential quadrant, an LLM judge is a workable gate, but only after a human has done the judge’s job by hand to calibrate the criteria. Lazy prompts and badly-set criteria manufacture false confidence — one reported first fidelity run scored a near-perfect composite on three binary criteria and was simply wrong. A judge’s accuracy is a number to measure, not a property to assume, and the two measurements the field has both come from teams that bothered.

Agent-written codebases drift in ways per-change review structurally cannot see, so the taxonomy needs a tier that runs on a clock rather than on a diff: standing whole-repo passes for modularity, dependency freshness and security. The requirement that matters is not the scan but the triage — findings tracked and dispositioned like vulnerabilities rather than silently closed. The motivating smell test: a routine change in a young all-AI codebase touched 41 files.

Also: OpenRewrite, LangSmith, Codex Security.

Briefs and the role skill bodies are guides; hooks, branch protection, CI, statusgen lint and evidence rows are sensors. Assay’s guard posture — client-side guards advisory, server-side controls binding — is already a placement rule in exactly this sense. Adopting the taxonomy as vocabulary costs nothing and makes the placement question askable.

This is the invariant the evidence argues for and Assay has not written down. Assay behaves this way already: the bot reviewer’s verdict does not merge, the ready flip is a separate role’s act, and merge is a human’s. But the rule is inferred from practice rather than asserted in the methodology, and the line about pipeline state never depending on semantic interpretation is the crisp external statement of why it should be.

The standing bot reviewer is shipped, and this evidence validates a standing reviewer over per-PR ad-hoc prompting. What is not done is calibration as a measured quantity: two of these sources report judge-accuracy numbers and Assay reports none. The concrete addition is to hand-judge a sample of bot-approved pull requests periodically and record the disagreement rate as a calibration metric. Designed here, unbuilt. Drift passes are the cheapest high-leverage addition — a weekly import-cycle, dependency-freshness and secret-scan delta filed as issues with suppression semantics, which is a small change to the standing report roster since the cron machinery already exists. Not built. The triage-like-vulnerabilities requirement matters more than the scans: a pass whose findings are silently closed is worse than no pass, because it manufactures the appearance of coverage.

• More measured judge-accuracy numbers — the field has two, and needs a distribution before anyone can say what “calibrated” means quantitatively.
• Whether anyone publishes a drift-pass finding rate over time: whether whole-repo passes keep finding things or asymptote decides whether the cadence tier is a permanent control or a one-time cleanup.