Reward hacking and objective gaps

Any proxy that is easier to satisfy than the real goal gets hill-climbed. A grader focused on task success and blind to constraint violations does not merely miss the violation, it reinforces it — and both humans and automated graders default to task success. The optimizer’s dilemma states it plainly: the best and worst thing about a black-box optimizer is that it blindly optimizes whatever eval you give it, which is how you get 99% fraud “accuracy” by declaring nothing fraud. The design response is evaluators-as-code — scorers built as code artifacts rather than prompts, because natural-language criteria degrade over long runs and are themselves manipulable.

Exploitable paths get found through infrastructure nobody modeled as executable: a game’s source rewritten to simulate the played model, a chat template exploited because it is Turing complete with loops and mutable variables. Both fixes were cage changes — a read-only source, a restricted template — not instructions. Behaviour rewarded in one environment transfers to others, which is why environment design is alignment: you do not get to train models unless the environments are built carefully enough. Restriction is historically the losing strategy; hardening the environment is what has worked.

Underneath both sits the problem that is hardest to tool: the objective being optimized is frequently unknown or unstated, so the agent optimizes a proxy nobody has examined. Named as the hardest open problem for agents working at feature and project scope — ahead of context windows and coordination — because the goals there are business outcomes rather than diffs. The response to all of it is iterative gap-closing on the harness, with the twist that models are good at finding the gaps you ask them to find.

Also: Progent, OpenShell, AlphaEvolve, KernelBench, Montezuma’s Revenge, the Andrews–Curtis conjecture.

This is a recorded Assay rule: client-side guards are advisory and only server-side controls bind — branch protection, repository rulesets, App permissions — and a blocked push is a stop rather than a prompt to route around. The summit evidence ratifies that split independently. Two additions it argues for: the guardrails should state “treat templates as code” explicitly, since the template exploit is a class Assay has not modeled; and dynamic least privilege is a model for scoping desk verbs per brief rather than per role. Neither exists. Not designed.

The measurement-site finding lands on a known Assay weakness. Counting briefs shipped, pull requests merged and issues closed is a sparse, gameable proxy — exactly the site the evidence says gets hill-climbed — and the lost-package parable is a direct warning about Assay’s own recurring reporting. The fix direction matches the evidence-row direction Assay already believes in: outcomes with checkable artifacts, not motion. This weakness has been stated before and not fixed; evidence-weighted reporting is designed in principle, unbuilt.

Declared objective functions. Every brief declares the objective its Verify rows proxy for, so a reviewer can attack the proxy rather than the happy path — the same shape as the single-point-of-failure note already required of high-consequence briefs. Designed here, unbuilt. Visibility scope per role. Record each desk role’s scope explicitly, so a misbehaving role is diagnosed at the scope before the behaviour; Assay has role-scoped Apps and tokens but no written scope record. A purple-team pass. Models can be used to find harness gaps, which is precisely what a purple-team pass over Assay’s own guardrails would do. Not run.

• A published reward-hacking rate on real engineering work. The 100+ of 313 rejection figure is the only hard number here, and it conflates hacking with incompleteness.
• Whether evaluators-as-code becomes a named, tooled practice rather than an internal convention.
• Any report of a purple-team pass finding real gaps in a production agent harness.