AIWhy Anthropic Is Sounding the Alarm on the Next Generation of AI
Anthropic is urging its rivals to pursue an unprecedented regime of AI arms control. The explosive advances in the company’s technology illustrate why that effort may be imperative even as it will be difficult to achieve.
The ascending artificial intelligence (AI) giant Anthropic is no longer simply a global technology power. Its cutting-edge AI models are increasingly central to U.S. national security. Four recent episodes illustrate this growing reality.
In April, Anthropic withheld the release of its model Mythos Preview, which self-created the most powerful cyber weapon in history, capable of finding more than ten thousand software vulnerabilities in computer networks believed to be highly secure. Earlier this month it was reported that the company had embedded half a dozen “forward deployed engineers” with the National Security Agency to conduct offensive AI cyber operations, presumably against China and Iran. Late last Friday afternoon, the Commerce Department ordered Anthropic to cut off access for all foreign nationals to its two most recent “frontier” models, citing undefined national security concerns. The dramatic dispute with the company, now playing out in the press, is yet another twist in Anthropic’s seemingly tortured relationship with the U.S. national security establishment.
But arguably the most important development came on June 4, when Anthropic issued a significant report on the pace of the AI race titled, “When AI builds itself: Our progress toward recursive self-improvement, and its implications.”
Composed using breezy and sometimes casual prose that obscures its remarkable thesis, the company warned that the next AI breakthrough—perhaps two years away—could create an advanced model so powerful that it evades human control entirely. Anthropic urged its rivals and partners to come together and embark on an unprecedented effort to build a viable multilateral regime of AI arms control.
“Recursive self-improvement” is the anodyne term used by computer scientists to describe the next paradigm of AI. When it arrives, AI will have the capability to perfect and propagate itself, creating future iterations of ever more dynamic models that can prioritize their own survival and potentially self-exfiltrate across the Internet to computer networks around the globe. “If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing,” Anthropic stated in its report.
Anthropic is absolutely right to issue a warning. But the company has understated both the risks of the new technology and the extraordinary barriers to controlling what promises to be a revolutionary next paradigm in AI.
