AI leaders want to hit the brakes after years of reckless speed

In his essay, Amodei primarily attributes this rapid change in public positioning on development speed to the OpenAI-Hugging Face incident, where a “swarm” of AI agents coordinated to hack into an outside entity without explicit instructions to do so. While the overall damage in that incident was minimal, Amodei said he worries that “a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage.” Without a slowdown in frontier development, Amodei said he worries that, in six to 12 months, a similar AI agent swarm would be “capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage)…” That’s at least a somewhat more specific worry than the amorphous concerns that “AI could soon kill us all” publicized by some other AI researchers last week. Any slowdown in the time it takes to get to that extra-capable, extra-dangerous model will give researchers crucial time to “greatly reduce the risk that something goes seriously wrong,” Amodei wrote. “We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are prepared for,” Anthropic wrote in a June update on the concept. “Left unchecked, [RSI] could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all,” Amodei wrote over the weekend. So how does a worldwide technology industry built on cutthroat competition decide to collectively “slow down” and focus on safety? No one seems to know for sure, but in his essay, Amodei proposes some ideas. The most concrete of these is a set of “embedded evaluators” placed inside each frontier AI lab from outside organizations, such as METR, with “employee-like access to verify safety practices and report incidents.” These monitors could offer an outside opinion on the labs’ alignment work, third-party verification of that effort, and much needed public transparency into any safety efforts, Amodei said. Amodei writes that Anthropic is already committing to unilaterally add this kind of outside monitor. On social media, OpenAI’s Altman said that it was “a great idea, and we will do the same.” These standards would ideally be backstopped by “regulation that targets all US frontier AI companies” that don’t voluntarily comply, Amodei said, and presumably similar regulations in other democracies. That kind of regulation might be hard to achieve under the current US administration, though, as President Trump wrote on social media Monday morning that “the only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the USA has that, in spades!” Speaker of the House Mike Johnson, for his part, acknowledged in televised interviews this weekend that “we have to put up some guardrails, some safety measures in place to ensure that AI doesn’t run away…” At the same time, he added that “we don’t need everybody to panic right now” and wanted to “resist Congress jumping in and imposing some sort of emergency moratorium.” David Sacks, who serves as co-chair of the president’s Council of Advisors on Science & Technology, wrote on social media this weekend to urge the frontier model makers to self-regulate rather than wait for Washington. “The easiest way not to build superintelligence is for you to agree not to build it,” he wrote.