you know what? it’s awesome that people are vigorously debating the credentials of assessors in the frontier AI industry. It’s awesome we are debating what independence really means. Some of it is in bad faith but who cares. This would have been my dream come true a year ago.
Totally agree. We need a variety of evaluators. But so long as evals, monitoring, and audits are voluntary, it’s not an attractive market to enter.
The most capable people have a lot of promising opportunities. This needs to outcompete their other options.
Right now, the only
We’ll need multiple, differentiated METRs.
But for that to happen there needs to be clear market incentives that make it worthwhile. Without government guaranteed and quality standards (1) we won’t get more METRs and (2) the ones that do emerge may have incentives to race to
This is a big deal. Dario and Elon are converging on something we’ve been pushing at Fathom - AI labs shouldn’t be the only ones judging whether their systems are safe. Independent Verification Organizations, or IVOs, can provide that outside check. Anthropic committing to embed
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our
