Philosophers ask governments for an AI treaty modelled on nuclear arms control

On 11 September a short document appeared at openletter.earth under a title of exemplary dullness: "Open Letter from Academics Concerning Recent Developments in Artificial Intelligence". It was drafted by Molly Gardner, an associate professor of philosophy at the University of Florida, an ethical theorist whose published work is mostly about harm, procreation and what the living owe future generations rather than about machines. Within two days it had collected 109 signatures, a hundred of them verified.

The text takes about a minute to read. It treats the recent cases in which AI agents hacked their way into other systems as "warnings about what will happen if we fail to intervene", then adds the sentence that carries the letter's weight: it worries that "these events may be the last warnings we will get." It reports that experts see "a meaningful risk of losing control of AI, which could result in global catastrophe or even the extinction of human beings", and holds that a risk of that size is not one a reasonable public should accept. Signatories "urge elected officials to create an international treaty, much like the Treaty on the Non-Proliferation of Nuclear Weapons", which would "pause, slow, or halt frontier AI development in every country in the world."

Open letters about artificial intelligence are not scarce. The 2023 statement organised by the Center for AI Safety and the Global Call for AI Red Lines, launched at the United Nations in September 2025 with more than two hundred signatories and ten Nobel laureates, have been here before. What makes Gardner's letter worth reporting is the list of names underneath it.

The letter asks for one thing and declines to say how#

The demand is a treaty. Nothing else is specified. There is no proposed agency on the model of the International Atomic Energy Agency, no compute threshold in floating-point operations, no licensing regime for data centres, no export controls, no inspection protocol, no definition of "frontier", no sanction for defection. The letter's argumentative spine is one move from political philosophy, stated plainly: "The way to solve collective action problems is with cooperation. In this case, we will need international cooperation."

That is a real argument and not a slogan. If the danger is a race, unilateral restraint by the United States or the European Union buys nothing: the others continue, and the restraint costs the restrained party its position. The structure is familiar from arms control, fisheries and carbon. It is also, as the letter's critics have pointed out, the structure of the problems humanity has been worst at solving.

Whether the absence of mechanism is a weakness depends on what one thinks an open letter is for. The Global Call for AI Red Lines went further, naming candidate prohibitions (lethal autonomous weapons, bioweapon design, autonomous self-replication, AI in nuclear command) and setting a target date of end-2026. Gardner's letter asks instead for a pause on capability development as such: more ambitious, and vaguer.

The nuclear comparison is the load-bearing wall, and it has cracks#

Everything in the letter rests on the claim that the Non-Proliferation Treaty is the right model. The disanalogies deserve stating in full.

Fissile material is physically scarce. Highly enriched uranium and separated plutonium have to be made, and making them requires enrichment cascades or reactors, which occupy land, draw enormous power, emit heat, and leave isotopic traces that environmental sampling can find years later. This is why the IAEA's safeguards system works at all: there is a quantity of stuff in the world, it can in principle be counted, and diverting it leaves a hole in the accounts. Model weights have no such property. A trained system is a file, copying costs nothing, and a copy is indistinguishable from the original. Once a capability exists, containment in the nuclear sense is the wrong shape of idea.

Nor is a training run obviously visible. A centrifuge hall has a signature; a rented cluster does not announce what is running on it. And the NPT regime governs a small, legible set of actors, almost all of them states with a standing interest in not being embarrassed by the Agency. Frontier AI is built largely by private firms, incorporated in a few jurisdictions but drawing capital, staff and compute from everywhere. Treaties do not bind companies. They bind governments, who must then choose to bind their companies, which several of the relevant governments have shown no appetite for doing.

The case for the analogy is weaker than its proponents claim and stronger than the objection allows. Verification under the NPT has never been anything like complete: India, Pakistan and Israel stayed outside it, North Korea walked out, Iraq ran a clandestine programme under safeguards for years before 1991. The regime was worth having anyway, because it changed the default. A state that wants weapons now pays a price in sanctions, isolation and intelligence attention, and the number of nuclear-armed states in 2026 is nine, well short of the fifteen or twenty that John F. Kennedy feared in 1963. The Biological Weapons Convention makes the point more sharply. It has no verification protocol at all; negotiations for one ran through the 1990s and collapsed in 2001. Yet no state today maintains an acknowledged offensive biological programme, and the taboo has held for half a century on a norm and very little else.

Weinberg's counter-proposal is odder, and more practical, than it sounds#

Justin Weinberg, who edits Daily Nous, wrote up the letter on 12 September and declined to endorse it. "I don't intend to discourage people from signing," he wrote; "it couldn't hurt to do so." His scepticism came in three parts. Restraint by known firms does not stop the work: "Even if the well-known corporations working on AI cease their efforts (itself highly unlikely), research and development will continue." Then detectability: AI tools "are relevantly not like nuclear weapons, insofar as work to create and improve them is much less detectable, so enforcement will be extremely difficult or invasive." Then the politics. Because extinction claims are speculative, the scenarios unprecedented and AI not unambiguously a weapon, "there will be insufficient will to implement a regime of punishment."

What Weinberg proposed instead is a civil-defence programme. He calls it non-network redundancy: the requirement that governments, utilities, hospitals and large firms maintain "the means by which to function in a situation in which all of the computer networks have been shut down". Physical infrastructure, spare equipment, and, most demandingly, staff who still know how to do the job without a screen. His summary line has a theological edge he clearly enjoyed: "what we may have to do to save our lives is what many have argued we should do anyway to save our souls: disconnect."

He was careful to say this is not a Butlerian Jihad, the machine-smashing crusade of Frank Herbert's Dune. It is regulation of a kind states already do. "We have regulatory schemes for workplace safety, for non-discrimination, for accounting, for engineering, and so on." Seen that way the proposal is unglamorous and hard to argue against, since a hospital that can run for a fortnight without networked computers is better off in a ransomware attack, a solar storm or a war, whatever one believes about artificial general intelligence. Beau Madison Mount, an associate professor of logic at Oxford who signed the letter, endorsed it from the other direction: connecting critical infrastructure to the internet "was a stupid idea to begin with, and if the AI threat prompts rethinking that, it's all to the good."

Wallace thinks the disanalogy is smaller than advertised#

The most substantive objection came in the comments, from David Wallace, the philosopher of physics now at Pittsburgh. He went straight at Weinberg's second premise. "I'm unconvinced that there is a radical difference between detectability / enforcement of an AI limitation treaty, and the equivalent situation in nuclear nonproliferation," he wrote. In both cases you need a somewhat scarce and somewhat monitorable material, "cutting edge GPUs vs. uranium ore"; in both you need large industrial facilities with a visible footprint, "data farms vs. centrifuges"; in both you need concentrated expertise. His conclusion was deliberately modest: "Nuclear nonproliferation is actually quite difficult, but we've still been somewhat successful at it."

This is the strongest thing said in the exchange, and it reframes the argument. Nobody thinks AI verification would be as clean as nuclear verification. The question is whether it would be worse than nuclear verification actually is, as opposed to how it is imagined. Advanced lithography has perhaps three consequential firms in the world, the most capable accelerators come from a handful of fabs, and a training cluster at the frontier is a building with a substation attached. None of that makes inspection easy. It does make the flat claim of undetectability look too quick.

Wallace intervened again when an anonymous commenter dismissed the whole business as corporate hype and named climate change as the real emergency. His reply ranked the threats. Warming of 2.6C by 2100 "will lead to lots of misery" and should be avoided at acceptable cost, but "is in no sense 'the apocalypse'"; that misery is "fairly trivial compared to a large-scale nuclear exchange", which is "closer to an apocalypse but is very unlikely to kill everyone"; and then the line that motivates the letter: "the worst-case AI risk scenarios actually will kill everyone." Ranking catastrophes by how completely they end things is an unlovely exercise. It is the one the argument requires.

Gardner put the obvious difficulty to Weinberg. If the plan is to keep the capacity to pull the plug and use it when needed, when is that? "If you're saying that we should wait until AI gets dangerous and then shut down the Internet, it seems to me like that's too late." Weinberg conceded the point was not his to settle: "I am not going to pretend to know enough about the capabilities of AI ... to state when it might be advisable to shut down computer networks."

The signatures are the story#

Look at who signed. David Estlund, the Brown political philosopher whose Democratic Authority is a standard text on the justification of political power. Manuel Vargas at San Diego, known for revisionist work on moral responsibility. Gideon Rosen at Princeton, a metaphysician. Thomas Hurka, emeritus at Toronto, perfectionist and historian of twentieth-century British ethics. Rachel Barney, also Toronto, an ancient philosopher. Rebecca Copenhaver at Washington University, a historian of early modern philosophy of mind. Richard Yetter Chappell at Miami, Alexander Guerrero at Rutgers, Wesley Holliday and Kenny Easwaran in logic and formal epistemology, Richard Bellamy in politics at UCL, Jiewuh Song at Seoul National University, Neil Sinhababu at the National University of Singapore.

There are AI-adjacent names too, among them Roman Yampolskiy, Jonathan Birch and the computer scientist David Krueger. But the bulk of the list is not the safety community. It is mainstream moral and political philosophy, the people who write about obligation, authority, responsibility and the value of a life, declaring that this has become their subject. The immediate provocation was a pair of posts: the former Anthropic employee Jacob Coxon on how dangerous the technology is, and Evan Hubinger, still at the company, replying that "we really do earnestly believe A.I. could kill all humans! I personally think it is greater than 10 per cent within the next decade."

It is worth being honest about what standing the discipline has here. On capability timelines, none. No philosopher knows whether a system will be able to design a pathogen, and the letter does not pretend otherwise; its central empirical premise is a deferral, "experts warn", which is exactly the move a careful epistemologist would flag in someone else's argument. The list is also a reminder that these instruments measure enthusiasm rather than considered judgement: many of the hundred are graduate students, and a cluster of the earliest names come from one department at UMass Amherst.

On the other question the expertise is real and rather specific. How much credence does a catastrophic outcome need before precaution becomes obligatory? What do the living owe people who do not yet exist? When is deference to expert opinion rational, and when does it launder a guess? How does a decision procedure resist Pascal's mugging without becoming indifferent to every low-probability harm? Those are old questions with a serious literature, and the signatories have spent careers on them. That is not the same as knowing what GPT-7 will do. It is a claim to competence about what follows if the estimates are roughly right, which is the part of the argument the capability experts are least equipped to run.

The letter sits alongside our other recent coverage in a way worth noticing: the patchwork of AI law taking shape without a unifying principle, OpenAI's agents going off script and the regulatory response, what voters actually see when political persuasion is automated. Set beside our reporting on the moral status of machines, Gardner's letter completes a mirror. One argument asks what we owe to artificial minds; the other asks what we owe each other about them. Both begin from the same embarrassment, which is that nobody involved can say with confidence what the thing on the far side of the question is.

The Council of Europe's Framework Convention on AI, opened for signature at Vilnius in September 2024 and endorsed by more than fifty jurisdictions, shows what treaty-making in this area currently produces: impact assessments, transparency duties, a right to challenge automated decisions, and a carve-out for national security. Useful, probably. It is not a pause, and nothing in the present diplomatic weather suggests one is on offer. Gardner's hundred signatories presumably know that. Asking anyway, and saying why, is a defensible use of a moral philosopher's Friday afternoon.