Harness engineering as a discipline

The core claim is that models outrun their own reliability. A frontier model can reason its way to the right answer far more often than a system built around it can be trusted to act on the world without supervision. That gap is not a model problem waiting on the next release — it is deterministic software work: marshaling, type checking, rule-based policy, deterministic tools wrapped around a probabilistic core. Name the layer, and it becomes something you can staff, test and version like any other component.

The field now has numbers, if not yet replicated ones. Swap the harness around a fixed model and scores, cost and latency all move — an 18% spread between best and worst harness for one open-weight model, a model scoring roughly 8% higher and $20 cheaper in a minimal harness than in its own vendor CLI, another failing to run at all inside a harness that lifts its peers. The blunt summary from one of those teams: the model is not the agent. The corollary is that a benchmark score quoted without naming the harness is close to meaningless.

Because the harness sits between intent and effect, it is the natural home of a control boundary. Hooks that can edit or block any prompt or tool call turn a convention into a wall: one team blocks pull-request creation unless the relevant skill was loaded, another blocks commits below a coverage bar. The phrasing that travels is that the model does not get to choose. That is a stronger claim than instruction-following — it is the difference between discouraging a mistake and making it structurally unavailable.

Against both of the above sits an absorption law: each model generation eats the previous generation's scaffolding, harnesses simplify, and then more complicated scenarios get defined and more complicated harnesses get built. So the durable asset is what accumulates around the harness — context corpora, evals, process, permission structure — not any one harness generation. Anything a coding agent can replicate in three months is not a moat. The frontier's open architectural question is whether the answer to harness churn is a layer above the harness, owning sessions, policy and routing while any harness executes underneath, or agents that construct their own scaffolding at runtime.

Also: Terminal-Bench; Omnigent; Imbue Manager; Hermes Agent; PyHarness; OpenHands; OpenSage; CyberGym; Cybench Pro; Xagentic; the Open Agent Leaderboard and its 10K+ published agent traces; Pi.

The desk layer is a harness in exactly this sense: narrow verbs instead of an open tool surface, curated context per role, client-side hooks backed by server-side branch protection and App permissions — advisory locally, binding on the forge. The standing rule that a blocked push or guard refusal is a STOP, not a prompt to route around is the same move as blocking pull-request creation unless the right skill loaded: the model does not get to choose. Assay also went looking at the competition first-hand, reviewing several third-party coding harnesses against its own before committing to the shape.

Each desk role has one neutral body with project-specific values resolved from the project layer rather than inlined, and a standing workstream exists to keep the desks runnable outside any one vendor’s harness. The model-independence argument and the abandon-the-custom-framework result both back that bet. The CLI-versus-MCP null result sharpens it into a constraint: a desk verb must be justified as a control plane — identity, authority, audit — and never as a usability layer, because interface ergonomics alone do not move capable models.

First, the absorption law and the three-month replication test argue for spending on the layers a model cannot absorb — brief and lifecycle discipline, human-gated merge, identity and permission structure, the evidence trail — and for treating the verb surface as consciously perishable. Second, the pair-plateau claim plus the measured 18% harness spread mean a desk change is a behavioural change that should be evaluated as a model-plus-harness pair; nothing in Assay measures that today. Third, the meta-harness position is the closest external analog to the desks, arrived at from the opposite direction — harness plurality first and governance later, where Assay is governance first over a single harness family — and it carries its own warning, that stickiness without openness is lock-in.

Assay’s answer is structure at the governance layer and freedom inside the brief. A worker may do anything its verify rows allow, but it may not redefine its role or mint new verbs. Worth watching whether models trained to build agents make a governed middle path — agent proposes a new skill by pull request, human gates it — safe.

• Whether anyone publishes a reproducible harness-ablation number on a fixed model. The 18% spread, the ~8%/$20 delta and the pair-plateau claim are the field’s only hard measurements, and none has been independently replicated.
• Whether the absorption law holds through the next model generation — which specific harness features get eaten, and whether governance-layer machinery (permissions, gates, evidence) survives as predicted.
• Whether meta-harness control planes become the standard place to enforce policy, or stay a single vendor’s answer to its own internal sprawl.