Essays, postmortems, and field reports on pre-registration, the PRML specification, and the wider question of what it would take for ML evaluation claims to be falsifiable in practice. New posts on no schedule. RSS.
The complete official documentation of a widely used evaluation framework — 165 files. None records what would count as passing, and none carries a mechanism for checking that the criteria came first. Its own experiment loop places the hypothesis after the results. Per-match ledger included.
29 published case studies from Singapore's Global AI Assurance Sandbox — a government-run programme that asks participants in writing to publish their interpretation thresholds. A reader can recover the pass rule in 3 of 29. Only one case states an explicit numeric threshold. Per-document ledger included.
Tampering attempts in about half of agent runs; eight benchmarks broken without solving a task; leaderboard access games. One study shows locking the evaluator eliminates tampering. The remaining question: prove the lock predates the run.
The Act tells you to declare your accuracy numbers and keep records that let them be checked. It does not tell you which artifact satisfies that. A practical comparison of model cards and pre-registered manifests, and the one property that separates them.
Lock #2 resolved at 0 / 3 tonight. Zero independent contributors filed RFC engagement during the two-week window. The mechanism worked; the outreach didn't. Plus the counting bug we caught two days before resolve.
Archived: the comment window closed 2026-05-22 and v0.2 is frozen as an RFC. Five proposals were open for community comment; v0.1 stays the stable spec, and v0.2 is additive.
