Are We Ready for a ‘DeepSeek for Bioweapons’?
This new potential for AI systems to pose extreme risks differs from frontier AI systems to date. Previous leading models—largely developed at Western companies like Anthropic, Google DeepMind, and OpenAI—have been cleared by their testing teams as not capable enough to be used for extreme harms. (Notably, some have argued that even those previous test results were not comprehensive enough to in fact establish a lack of risk.) Anthropic has now concluded that extreme harms are possible: In an interview, Anthropic’s chief scientist gave the example of strengthening a novice terrorist like the Oklahoma City bomber or helping an amateur synthesize a more dangerous flu virus. Specific tests described in Anthropic’s system card include whether their model could help people with only a basic background (for example, undergraduate STEM studies) make “a detailed end-to-end plan for how to synthesize a biological weapon,” or whether the model can function as an expert in your pocket for answering sensitive related questions.
Adler notes that
Anthropic’s announcement that their AI system has triggered this new risk level carries three important implications. First, Anthropic crossing this threshold suggests that many other AI developers will soon follow suit because the means of training such a system is accessible enough and well understood. Second, other Western developers cannot be counted upon to take the same level of precautions as Anthropic did—either in testing or in applying risk mitigations to their system—because, in the absence of federal or state safety mandates, society is relying on purely voluntary safety practices. Third, the international scale of anti-proliferation for powerful AI systems will require even more than just domestic safety testing regulation (though that would be a good start). The world isn’t yet ready to head off the risks of these systems, and it might be running out of time.
He stresses that we need to be prepared for many more groups to imminently develop similar systems. “Anthropic does not have some secret technique that allowed them to train a model this capable; it’s just a matter of time before other AI developers (first just a few, then considerably more) can create a model that is similarly capable.”
We need to be ready, but the trouble is that safety practices are voluntary, and not all developers who create such a powerful system will take the precautions that Anthropic says it has. Some U.S. laws have tried to require certain safety practices, but none have succeeded.
Adler concludes:
I’m using DeepSeek as the example—though such a model could be developed by other groups as well—because it has three attributes that increase the risk of misuse: First, it is freely downloadable to anyone; second, it is impossible to enforce safety mitigations upon; and third, it is developed outside of U.S. jurisdiction. This freely downloadable approach—sometimes called “open source,” or more appropriately, “open weights”—is in contrast to Anthropic’s approach of taking significant steps to prevent theft by adversaries like terrorist groups. Because the model would be freely downloadable on the internet, there is no permanent way to apply safety limitations to prevent users from obtaining help from the model with regard to bioweapons-related tasks. And being outside U.S. jurisdiction will limit the U.S.’s influence, even if it does eventually pass AI safety regulation.
It is possible that Anthropic is mistaken about the risk of Claude Opus 4, meaning that a company like DeepSeek matching its capabilities would not in fact be that risky. Not many people want to actually harm others with bioweapons, even if they suddenly have stronger means of doing so. Moreover, it could be that acquiring the necessary lab materials—not just improving one’s scientific know-how—proves to be more of a bottleneck than believed. (Anthropic has considered this bottleneck, however: Acquiring useful materials related to bioweapons is one example of an evaluation conducted in the risk determination.)
I do not find it especially likely or comforting, however, to simply assume that Anthropic’s risk assessment is mistaken. Instead, we need to recognize the collision course ahead: It seems there will soon be widely accessible AI systems that can help ordinary people to develop dangerous bioweapons. Perhaps the AI systems will not excel at every single part of the workflows for causing these harms—acquiring raw materials, synthesizing a substance, developing a plan to release it—but the risks are still meaningful. Some of these systems will be developed outside of U.S. jurisdiction, which limits the U.S.’s influence. Other countries, like China, will need to grapple with the same reality, in terms of being unable to control what powerful systems the U.S. develops or releases. Given the national security dynamics at play, how does this end?
For the world to manage powerful AI safely, we need at least two things: first, to figure out sufficiently safe practices for managing a powerful AI system (for example, to prevent catastrophic misuses like terrorists synthesizing a novel bioweapon); and second, to ensure universal adoption of these practices by all relevant developers—“the adoption problem”—not just those within the U.S.’s borders.
Domestically, we need a legally mandated testing regime to even know what models are strong enough to demand mitigations. Features of such frontier AI regulation should include clear specifications of what models need to be tested, based on objective inputs like the amount of compute or data that went into creating the model. Otherwise, it may be left to developers’ discretion to determine what models are considered “frontier” and therefore subject (or not) to elevated testing. Moreover, certain aspects of the testing regime should be mandated as well to reduce the competitive incentive to cut corners. For instance, perhaps there should be a “minimum testing period” for the leading frontier AI systems, to ensure that their developers have adequate time to test for concerning abilities. Testing alone certainly isn’t sufficient; the AI industry still needs “control” techniques for reducing the risk posed by a dangerously capable model, among other interventions. But the lack of mandatory testing and safety standards in frontier AI today is in stark contrast to how the U.S. approaches other safety-critical industries, like aviation.
Internationally, the challenge is admittedly tough but tractable. Today the U.S. is pursuing the wrong strategy. “Winning the AI race” misframes the point—we need mutual containment, not a race to dangerous capabilities. As one senator recently put it, “If [there are] gonna be killer robots, I’d rather they be American killer robots than Chinese.” But developing American killer robots wouldn’t prevent the creation of Chinese killer robots shortly thereafter. Getting to some level of AI capability first—the “racing” approach—is not a sufficient strategy for the U.S. Yes, U.S.-China relations are strained, and surely the U.S.’s recent tariffs don’t help. But cooperation serves both nations’ interests—not just heading off a threat posed by the other, but also preventing bioweapons from falling into terrorists’ hands. We’ve negotiated chemical weapons bans before; AI treaties are possible.
And if we don’t take action on this soon—coming to agreements between the major powers of the world about how AI will be developed and used, and what abilities it will be permitted to have—we need to be prepared for the consequences: like a freely downloadable “DeepSeek for bioweapons,” available across the internet, loadable to the computer of any amateur scientist who wishes to cause mass harm. With Anthropic’s Claude Opus 4 having finally triggered this level of safety risk, the clock is now ticking.
