The harness feedback flywheel

The slogan version is easy and the practice is not. The industry’s predominant mistake, as one talk named it, is correcting the agent’s output instead of improving the system that produced it — “it works with my agent” being the new “it works on my machine”. The discipline is back-propagation without gradients: watch the bad run, find where the agent failed, and push that back into the provisioned environment so the next attempt pulls the right context just in time. Each decomposition permanently accrues reliability.

What separates the practice from the slogan is a ladder of increasingly expensive fixes: re-prompt, then write it down, then have a reviewer agent judge every diff, then a static lint or guardrail. Counterintuitively the recommendation is to intervene as far right as possible, minimizing synchronous human time, and to climb leftward only when recurrence proves it necessary — because promoting every one-off into a permanent rule is exactly how instruction files rot and start competing for attention with the task.

A fix has to land in the layer that actually catches it. A rule that fires weekly belongs in a lint message that points at the right API, not in a context file. The complementary move is deterministic where possible: turn conventions into constraints, run the loop end to end once, and treat the breakage as the specification for the next constraint. Turn linting up, on the reasoning that generated output needs it more than handwritten code, not less — a bad system just gets bad faster.

Because harness edits are behavioural changes, the loop runs backwards too. A model swap or harness upgrade silently changes behaviour the way a breaking external API change does, so a version bump needs a frozen eval set gating it, and any benchmark quoted without a pinned harness is close to meaningless — swapping the harness around one model has been reported to move scores by up to 100%. The field is beginning to name skill portability as an eval question: does this skill survive a model swap?

Also: DORA; Adam Tornhill’s Code Health.

The loop is already canon. When a worker setting its git identity inside a linked worktree clobbered the shared configuration for every other session, the incident became a written rule plus a hook — the ladder climbed exactly as the field describes it. The findings register is the durable record of that class of promotion, and the rule to cite a finding by link rather than restate it is what keeps a promoted rule single-homed. “File at discovery” is the same instinct pointed at systemic issues rather than harness edits.

First, every guard refusal message should be self-service agent-facing prompt text — the rule violated, the canonical fix, a link to the finding — so the refusal itself climbs the ladder instead of ending the session. Second, repeated reviewer findings should be ranked into an explicit promotion queue rather than promoted by whoever happens to notice; recurrence is the field’s only sanctioned promotion signal, and Assay currently estimates it from memory. Third, “file at discovery” could be a wrapper that auto-attaches version, repro and a duplicate check, rather than a discipline each session re-performs.

Assay pins its tool versions and has a release chain for bumping them, but it has no behavioural gate on a model or harness bump: nothing today proves an upgraded desk skill still does what it did. The obvious shape — designed, unbuilt — is one fixture task per desk skill, run headless on upgrade, with the pin bump gated on it, treating a harness or model bump as a behavioural release. The missing specification detail for anything more automatic comes from the enterprise worked example: a recommended change must arrive carrying regression evidence, not just a diagnosis.

Worth stating so the two are not confused: the unbuilt dreaming pass is this concept’s outer loop, not a memory tidier. It is a requirements-, model- and evaluator-revision engine over deployment surprises — failed verify rows, review findings, incident notes — which is the same second loop argued as structural above.

• Whether anyone ships the drift gate end to end: a frozen eval set that actually blocks a model or harness version bump in CI, rather than a benchmark run after the fact.
• Whether the 19-configuration harness-sensitivity paper lands with reproducible numbers; it would be the first hard measurement of the up-to-100% claim.
• Whether auto-generated harness PRs, or recommendations that carry their own eval evidence, get adopted anywhere a third party can audit — which is what would move automation of this loop from vendor claim to practice.