Skill eval gates

The shape the field settled on maps straight onto software testing. Unit-level evals exercise one skill in isolation, cheap enough to run against small models to catch regressions. Integration-level evals run a whole installed set over a repo, with scenarios extracted from old pull requests. End-to-end scenario evals check the system as a whole. As with tests, scenario quality is the thing that decides whether the tier is worth anything, and the gate belongs in CI: content, format and trigger checks that block a merge, not a reviewer reading prose.

Underneath sits a harder measurement problem. In the largest study in this scan — 500 skills across 1,000 synthetic tasks and 19 model/harness configurations — goal completion sat at 90–93% with or without the skill present. Outcome alone therefore measures almost nothing. The usable signal decomposes into three: activation (did the skill fire at all), trajectory (did the run follow the intended steps), and outcome — with the skill's own text serving as the rubric the trajectory judge scores against. Structural checks catch none of this: one deliberately absurd skill passed front-matter and line-count linting and still scored 20% from a judge.

That implies a baseline discipline. With no skill present, agents still followed the skill's instructions 55% of the time — so an instruction the model would have obeyed anyway is pure attention competition and should be deleted. Lift concentrates in library and API choices, conventions, required steps, prohibited or deprecated patterns, house style, and security and compliance rules; it is lowest for generic data processing and for testing, where models are trained heavily already. The clearest case is a CLI that shipped after every frontier training cutoff: only a skill fixes that at inference time, six to nine months before the weights catch up. The same measurement answers the retirement question — has the model got good enough that this skill is no longer needed?

Also: Tessl skill lint and review; LangSmith; JSONL eval datasets; OpenEnv's init and validate; PaperClip paper agents.

Assay verifies work product thoroughly — the Verify table on every brief, drained by a separate role after merge — and verifies its own skills not at all. The skill bundle ships on reviewer judgement plus a structural lint that checks shape and holds duplicated copies identical. It never asserts behaviour, and structural conformance is exactly the bar the absurd get-rich-quick skill cleared.

Black-box scenario evals come first, because the desk roles have deterministic observable outputs — which action a role takes, which gate it refuses at — so trajectory-level assertions are available without an LLM judge, and a fixture corpus can be built from real triaged inbound. Second, a judge pass added to the existing lint, advisory rather than blocking, so the signal exists before anyone argues about the gate. Third, a lift baseline before any rule earns its tokens: that study turns "should this rule be in the skill?" from taste into measurement, and predicts the answer here — the role vocabulary, the claims protocol and the trust-gate rules appear in no training corpus and should dominate, while generic software-craft passages should score near zero and be deleted. Fourth, an admission bar for the public bundle: no skill joins until a fresh agent, given only that skill, reproduces the claimed capability on a held-out task — plus the whole-system rule that a new skill must prove it does not regress the installed set.

• Whether anyone publishes an activation / trajectory / outcome eval harness that others can run, or whether every house keeps rebuilding one.
• Whether the 55% no-skill instruction-following baseline replicates outside that one corpus — if it does, it is the single most useful number in this theme.
• Whether auto-release-on-passing-evals spreads beyond one organization, which is the signal that skill evals are trusted rather than merely run.