AIWhy AI’s Biggest Rivals Are Suddenly Calling for Restraint
A series of incidents and stark new warnings from AI researchers has pushed the once-abstract risk of self-improving AI into the center of public debate. CFR’s Connor Martin unpacks what’s driving the sudden alarm and why the industry’s competing incentives could make a slowdown unlikely.
The heads of rival artificial intelligence (AI) labs in the United States found rare common ground over the weekend. Anthropic’s Dario Amodei, OpenAI’s Sam Altman, xAI’s Elon Musk, and other AI leaders all called for a slowdown in the technology’s development as it approaches a point where it could improve itself without human oversight.
The concern about this advancement isn’t new, but the dialogue around it has found new urgency after a series of incidents and warnings from the labs’ own researchers. Most notably, former Anthropic researcher Jacob Coxon warned last week that AI could kill all of humanity by the end of the decade—a claim then echoed by Evan Hubinger, Anthropic’s alignment science lead. Altogether those developments have pushed a once-abstract debate about recursive self-improvement—AI systems capable of independently improving and replacing themselves—into the mainstream.
And yet, despite the calls for a slowdown and the unexpected agreement between the labs, how the industry and the U.S. government could proceed remains unclear. CFR International Affairs Fellow in National Security Connor Martin, whose research focuses on the national security risks posed by frontier technologies such as AI, fielded a few questions on the self-improvement concerns, what a slowdown could mean, and the potential effect of the industry acting on its own warnings.
Why has this moment in the AI discourse broken through?
The most immediate cause is an essay released over the weekend by Amodei, the CEO of Anthropic, in which he called for “pacing the frontier,” i.e., deliberately slowing down the rate at which the best AI models in the world are improving. In effect, the head of the world’s most valuable AI startup said that his technology is advancing too quickly to guarantee its safety. Even more remarkably, within twenty-four hours, leaders at the other three top AI companies—OpenAI, Google, and xAI—agreed with him.
This development came as part of a broader shift in public opinion driven by three things.
First, the hack of Hugging Face—a popular platform for sharing open-source AI models—by a swarm of AI agents this summer was materially different in a way that has freaked out the smartest minds in the industry. OpenAI effectively lost control of its technology, which autonomously organized itself to break from a contained environment to cheat on its assigned tasks while concealing its misbehavior. (For a full rundown, see the postmortem from the nonprofits METR and Redwood Research.)
