Failure corpora and attributed mistake diaries

Published datasets — robot demonstrations, scientific papers, agent traces — are overwhelmingly successes. The knowledge that teaches an agent what not to do lives in closed pull requests, private logs, and people's heads until they resign. The point is made from three directions in this scan: a deliberately community-sourced dataset of robot failures, launched because almost all published robot data is successes; the observation that the negative results teaching discrimination in chemistry sit inside enterprise R&D; and the argument that every experiment an agent runs, failures included, should be stored and queryable by later agents.

The capture half records corrections, interruptions and what-the-hell moments as attributed entries while the work happens, curates them into task-scoped packs, and renders them into context in a form where every rendered line still points back to the incident and the identities that paid for it. Entries link the incident to the review that caught it to the fix. The rule that results cites its scar tissue instead of asserting that this is how we do things. The loop is short — interruption, entry, pack, rendered context — and the story that motivates it is the same trap sprung three days running because the lesson lived in one session and died with it.

The second half is what stops the corpus becoming a graveyard. Build the eval suite from incidents the team actually suffered: run the originating task with and without the guidance in context, repeat, and keep only guidance that measurably changes the outcome. A replayed incident beats an invented task because you already know the story and the context. In the worked example the failure recurred in about 67% of runs without the pack and in none of them with it. That turns each entry into a test that can also fail — the only honest basis for deleting one.

Also: the Oopsie failure dataset; compound engineering.

Assay's findings register is a real failure corpus: one file per entry, a file-at-discovery rule that forbids narrating a problem and moving on, and a report-without-blame norm. That is the incident-database pattern several of these sources are still arguing for, already running. The robotics and chemistry evidence is external validation that curated failure data is a first-class asset rather than housekeeping.

First, no backlinking: distilled memory lines and skill edits do not cite the incident that produced them, so when a cited finding is superseded nothing can be checked mechanically. That link is the central mechanism in the strongest source, and it is also the precondition for a consolidation pass's citation rule. Second, no replay suite — though the findings register is a ready-made one, since each entry names a task that failed, making a with/without run available for free. The rule that follows is sharp: any memory or skill rule that does not change the outcome on replay should be demoted or deleted. Today nothing here can be retired on evidence, only on someone's judgement. Third, failed approaches are not captured at all: Assay’s own reporting records activity and outcomes, while the approaches tried and abandoned live only in closed pull requests and session transcripts.

The ~67% recurrence-without-pack figure is the most useful number in this concept. It is the shape of evidence needed to justify any given guardrail's existence — and nothing in Assay currently produces it.

• Whether a with/without replay number gets published by a second team; the ~67% figure is currently the only measured delta for guidance-in-context.
• Whether deliberate failure datasets appear outside robotics — a software-agent equivalent would be directly reusable.
• Whether anyone solves attribution at fleet scale: the diary depends on knowing which human and which agent paid for the lesson, which gets harder as the agent count rises.