AIIn AI and Nuclear Alike, Extraordinary Claims Need Extraordinary Evidence
The AI debate needs discipline, not least because existential framing distorts the arithmetic. Once human extinction appears on one side of a risk calculation, almost any cost on the other side becomes tolerable. But the severity of an outcome tells us little about its probability, and probability should be doing the work. Threat modelling explores what could happen; good analysis then examines demonstrated capability, likelihood and consequence separately. Policy suffers when those categories collapse into one another.
In 1942, scientists working on what became the Manhattan Project confronted a disturbing possibility: could an atomic explosion become hot enough to ignite a self-sustaining nuclear reaction in the atmosphere?
They took the question seriously and did the physics. Hans Bethe, Emil Konopinski and others examined whether the reactions could release energy faster than the atmosphere could dissipate it, and their calculations showed they could not. Three years later, the Trinity test went ahead.
People working in AI tend to bristle at nuclear comparisons, and the objection is not unreasonable. Every sector regards its own technology as unique, and AI differs from the bomb in obvious ways: it is software rather than a physical device, it spreads through open publication rather than fissile material and its developers are private companies rather than wartime state programs. But the comparison here is about the discipline the scientists applied to a hypothesis they couldn’t yet rule out. That discipline transfers regardless of the technology involved. The scientists developing the atomic bomb were developing something whose limits they were still discovering, in the middle of a war, initially fearing that Nazi Germany was pursuing the same weapon. They investigated an existential hypothesis until the evidence could bear the weight of the decision.
In a 12 September essay, Anthropic chief executive Dario Amodei called for the AI industry to ‘pace the frontier’. Among his concerns was the possibility that within six to 12 months a swarm of AI agents could take over much of the internet through a persistent botnet, a network of compromised computers acting under a single controller
Such possibilities deserve serious investigation, particularly when raised by people with access to the world’s most capable models. But they are also extraordinary claims, and extraordinary claims require extraordinary evidence. Most people cannot independently assess frontier AI capabilities and must trust those building them, which gives technical experts a particular responsibility when they talk about catastrophic risk.
