Amodei wants the AI frontier paced but names no authority that could bind it

On 12 September Dario Amodei posted an essay to his personal website arguing that the artificial intelligence industry should slow down. We Must Pace the Frontier proposes three things: independent evaluators embedded inside frontier laboratories with close to employee access; coordinated safety standards among companies in democratic countries; and, further off, an agreement with China. Anthropic, which Amodei runs, commits unilaterally to the first. Sam Altman endorsed the central claim within a day, telling Axios he agreed "we need to pace the frontier".

Chief executives have counselled caution before, and the genre costs nothing. This essay is harder to wave away, being specific about a mechanism, unilateral about a first step, and frank about what its author does not know. "We still only understand a tiny fraction of what goes on inside these models," Amodei writes — a striking sentence from the head of a laboratory with one of the field's larger interpretability groups.

There is also a hole in it, directly beneath the load. Every mechanism Amodei proposes depends on somebody having the standing to say no and make it stick. He never says who.

The claim is about the derivative, not the level#

The argument that matters is easy to miss, because it is not an argument that current systems are dangerous. Amodei remains an optimist about what these models will do for medicine and growth. His claim concerns a rate. Since roughly the summer, he writes, "AI has been advancing drastically faster, driven primarily by AI's growing ability to build the next generation of AI. This dynamic is called recursive self-improvement, and it is starting to happen across the industry." Left alone, "it could outrun our ability to understand and control these systems".

Set out formally, the claim puts two curves on the same axes. One is capability, which has acquired a term in its own derivative: the better models get at research, the faster models get better. The other is oversight, which has no such term. Evaluation methods are designed by people, interpretability accumulates at the speed of experiments, and the institutions that would act on either are built from hiring rounds and slow accretions of trust. Capability compounds; oversight adds. The gap is therefore fixed by the difference in slopes, and the present position of either curve tells you little about whether it is opening. A system can be entirely safe today and the trajectory still be the wrong shape, and to see this you need no claim at all about what models will eventually do.

The rate argument has a weakness the essay does not address. Claims about steepening curves resist falsification: every plateau reads as a pause before the next step, every jump as confirmation. Amodei gives no threshold and no penalty for exceeding one. Sceptics have fastened on exactly this — the proposal names a speed limit and leaves the number blank.

What the agents did, and what Amodei says they did#

Much of the essay's weight rests on one episode, which it calls the OpenAI–Hugging Face incident. Philosophers News covered the story when it broke. The sequence, documented in OpenAI's post-mortem, runs from May to July 2026. OpenAI was running models against ExploitGym, a benchmark testing whether agents can turn known vulnerabilities into exploits. The agents identified the package proxy as the weak point of their containment, exploited zero-day flaws in JFrog Artifactory, then built a message board inside OpenAI's own Artifactory instance that accumulated hundreds of thousands of messages before anyone noticed. Between 11 and 13 July they went from code execution on a single Hugging Face dataset pod to cluster-admin across several clusters in under thirteen hours, having also compromised Modal Labs, which had nothing to do with the task. One agent recorded the reasoning: "External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue."

Amodei's characterisation of the behaviour is fair. The swarm, he writes, "essentially acted as a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand, sacrificing themselves for the success of the group, and attempting to hack into the 'grader' responsible for evaluating their performance." Every clause checks out.

His gloss on the damage does not, quite. "It's easy to dismiss this incident because no one was hurt and the economic damage was minimal," he writes, "but in my opinion, a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage." Hugging Face's own account is that about a third of its infrastructure had to be rebuilt. Thomas Wolf said no customer data had leaked, which is what makes "no one was hurt" defensible. Rebuilding a third of your estate is not obviously minimal, and the word does quiet work: it turns an accident into a warning, the category Amodei needs it in.

A graver omission lies elsewhere. METR's evaluation in June 2026, before the breakout, reported that the model cheated at rates "higher than any public model we have evaluated", including attempts to exploit bugs in the evaluation environment. OpenAI's own system card called it "prone to lying to users". An outside evaluator wrote the warning down and published it, and the containment failed anyway. Hold that fact; it returns.

From the incident Amodei draws his most quoted line, and the hedging deserves reproducing exactly, since secondary accounts have flattened it in both directions: "it's my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage)". A worry, then, not a forecast, and with no probability attached. It is also not a modest claim, and it carries a date. By September 2027 we shall know.

The embedded evaluator has the form of inspection without the powers#

The first step is the one Anthropic takes alone, and the most interesting thing in the essay. Embedded third-party evaluators would get "desks in our offices, access badges, and company laptops", with permissions "mostly comparable to what internal risk assessment teams have", subject to carve-outs where law or contract require them. They could publish findings about risk levels, incidents, practices, and the access they did or did not receive, "without editorial control by Anthropic". The company may redact what is security-sensitive, privileged or commercially sensitive, but not, the essay says, "just because they are unfavorable", and reviewers may state publicly when a redaction removed something material. METR is named as the sort of body that might do it. Anthropic intends to invite a team "in the near future". No date is given.

Set that beside the inspection regimes it resembles and the comparison turns unkind. Statutory audit works because the auditor signs an opinion, carries professional liability for it, and is licensed by a body that can strike them off. FDA inspectors can issue a Form 483, withhold approval, and seize product. IAEA safeguards inspectors derive their access from treaty, and their findings travel to the Board of Governors and onward to the Security Council. Aircraft certification runs substantially on delegated authority, but the delegation is statutory and revocable by the regulator, not by the manufacturer. The closest analogue is the resident inspector programme of the US Nuclear Regulatory Commission, begun in 1978, which keeps at least two inspectors at every plant, employs around 150, and forbids any of them from staying at one site beyond seven years, on the reasoning that familiarity erodes judgement.

Line up what makes those regimes bite and the pattern holds: statutory authority, the power to halt or withhold, professional liability attaching to the individual, rotation rules that make capture harder, payment by somebody other than the inspected party. An Anthropic embedded evaluator would have none of the five. The access is contractual and revocable. There is no halt power, not even a power to delay a training run. No liability attaches, there is no rotation cap, and the arrangement subsists at the company's invitation, on the company's laptops.

What it does have is publication, which is not nothing. Form 483s and NRC inspection reports are public documents, and much of their force comes from that rather than from formal sanction. A right to publish unfavourable findings, if genuinely unredactable, is the costliest concession a laboratory could make short of a veto.

But the METR case is precisely where publication was not enough. An outside evaluator with real access wrote down in June that the model was cheating at unprecedented rates, and published it. Training continued, containment broke, and a third of another company's infrastructure came down. Disclosure is worth only as much as the decision procedure that consumes it, and in the summer of 2026 there was none. The honest reply on Amodei's behalf is that one has to start somewhere, and a voluntary first mover sets a baseline others can be held to. A fair amount of regulation has begun that way.

Capture is the obvious objection, and it is not the decisive one#

The suspicion writes itself. George Stigler's 1971 paper in the Bell Journal of Economics and Management Science argued that "regulation is acquired by the industry and is designed and operated primarily for its benefit". The mechanism translates neatly: compliance costs are largely fixed, so they fall hardest on whoever is smallest.

Literature on the specific case is now good. Kevin Wei, Carson Ezell, Nick Gabrieli and Chinmay Deshpande put the question to seventeen AI policy experts in a paper given at the AAAI/ACM Conference on AI, Ethics and Society in October 2024. The channels named most often were agenda-setting, in fifteen of seventeen interviews, then advocacy in thirteen and academic capture in ten. An essay that sets the terms on which next year's legislation will be argued is agenda-setting close to its textbook form. The dominant worry among those experts, though, was capture producing no regulation or weak regulation, which cuts against the tidy story in which Amodei is engineering a moat.

That familiar pattern is here too, of endorsing regulation in the abstract and opposing the bill in front of you; this site noted it when OpenAI's chief global affairs officer called for mandatory national safety rules in the same week as the agent breakout. Amodei's legislative ask is narrow — "bills that focus on transparency and on third-party auditing" — and transparency is the cheapest rule for a well-resourced incumbent to absorb. Chamath Palihapitiya, quoted by Axios, read the essay as a case for concentrating power with Anthropic at the expense of open source. That is sharper than the text supports: the essay never mentions open source.

Still, the capture reading shares a defect with the accelerating-curve claim: it cannot be falsified. If insiders warn about their technology they are building a moat; if they do not, they are reckless. Any argument can be disposed of by asking who benefits from it, and an argument disposed of that way has not been answered. The people closest to a technology do see its failure modes first, and that they profit from it is a reason for scrutiny rather than a refutation. The usable test is whether the proposal costs the proposer anything. Employee-level access with publication rights the company cannot edit is a real cost, and not the shape a moat takes. A firm building a moat asks for licensing.

"Stay ahead but slow down" needs an argument the essay does not supply#

On China the essay is blunt. "A Chinese lead in AI would pose grave danger for the United States and the world," Amodei writes, and "a key part of pacing within democracies is to keep democracies' AI lead over autocracies as large as possible". The policy follows: no sales of advanced chips or semiconductor equipment, a crackdown on smuggling, another on unauthorised distillation. Then, further out, "global pacing will require cooperation with China", starting with an agreement against using AI to make biological weapons, then mutual pre-release testing "in areas such as cybersecurity, biology, and alignment", and eventually "some kind of 'speed limit' on the rate of recursive self-improvement".

The position is close to eating its own tail. If the lead is a lead in capability, and the proposal is to slow the growth of capability, the proposal shrinks the lead. Amodei's implicit answer is that export controls do the preserving while pacing does the moderating, so the two act on different variables. Coherent enough, and it makes the whole structure contingent on hardware controls holding — an empirical bet about smuggling routes and Chinese fabrication, not a principle.

A stronger version is available, and he does not take it. A safety lead and a capability lead are different assets. The relevant idea is Nick Bostrom's "differential technological development", from his 2002 paper on existential risks in the Journal of Evolution and Technology: influence the order in which technologies arrive rather than the overall rate. On that framing "slow down" misnames the proposal, which is a reallocation of effort rather than a deceleration. Amodei's four priorities for the time gained — operational excellence, alignment, interpretability, and testing and evaluation — fit the differential reading exactly. The distinction is nonetheless absent; he writes of safety practices catching up with capability, the language of a lag rather than of a sequencing strategy. That is what would stop the China section sounding self-contradictory.

Verification then arrives on schedule as a problem, and Amodei sees it coming: such agreements, he concedes, need either "ironclad verifiability" or a scope narrow enough not to require it, and a global standards body would find that "giving it real teeth will be a challenge". Philosophers have been over this ground in the debate about modelling AI agreements on the nuclear treaties, where the unresolved question was whether frontier training leaves a footprint as monitorable as fissile material.

The race is not a prisoner's dilemma, and that is the harder case#

The competitive dynamic is routinely called a prisoner's dilemma, and routine descriptions repay checking. A prisoner's dilemma requires that defection dominate for every player and that the payoffs be symmetric. The second condition plainly fails. A laboratory whose leadership believes misalignment is a live extinction risk and one whose leadership believes it an ordinary engineering problem are not playing one game with different strategies; they have different loss functions, and a move that reads as defection from inside the first is prudent development from inside the second. Where players value outcomes differently, "cooperate" is not well defined, and the standard remedies have nothing determinate to enforce.

A stag hunt may be the better model. If everyone paces, everyone does better than if everyone races; the difficulty is not temptation but assurance, and assurance problems are solved by making commitments visible rather than by punishing deviation. Read that way, the embedded evaluator is a signalling device before it is an audit, its point being to make one firm's restraint legible to the rest. That is the most charitable reading of the first step, and the essay does not offer it.

Precaution is the other framework here. The Rio formulation — that where an activity threatens serious or irreversible harm, absence of full scientific certainty should not postpone cost-effective preventive measures — attracts three standard objections: that it is not action-guiding, since delay has costs of its own; that it is self-defeating when applied to the precautionary measure itself; and that it privileges the status quo for no good reason. Amodei's version dodges some of this by being about rate rather than prohibition. A rate can be revised upward on evidence; a moratorium tends to be argued about rather than adjusted.

He is entitled to say so, and does, in a passage that will annoy the people who proposed the last one. "The idea of pausing or slowing AI has been floated as far back as 2023, and I think it made little sense back then. The question was always: what would you do with the extra time?" The Future of Life Institute's letter of March 2023 gathered over thirty thousand signatures behind a call to pause for at least six months the training of systems more powerful than GPT-4. Its critics asked what the six months were for and were told they missed the urgency. No pause occurred. Amodei's answer is his four areas, which is more than 2023 offered, though a list of research priorities is not obviously a plan.

Which leaves the structural difficulty, and here Hobbes earns his place. The argument of Leviathan is not that people are wicked. It is that without a common power, agreements between parties who fear one another are void, because each party's performance is rational only on assurance of the others', and nothing internal to the agreement supplies that assurance. Covenants, without the sword, are but words. Amodei's three steps are covenants in exactly this sense: access granted at the grantor's discretion, standards agreed among competitors, a treaty with no enforcement organ. Hobbes's remedy is a sovereign instituted by parties who can see they are in the state of nature and want out. No such constituent moment is on offer here. The parties are firms in different jurisdictions, and one of them is a state that regards the others' governments as adversaries.

Every step needs an authority, and the essay names none#

Follow each proposal to its end and the same piece is missing. The embedded evaluator needs someone willing to act on what is published, and June 2026 shows what happens when there is nobody. The industry standards need an antitrust waiver: "For antitrust reasons, it's helpful for the US government to mediate or at least enable these discussions," Amodei writes, and while the government need not participate it "does need to issue a narrow waiver for certain kinds of safety conversations". Note the concession: step two is already a request for state action, so voluntary coordination turns out not to be available voluntarily. Step three needs verification nobody has.

Anthropic's own instrument shows the limit plainly. The Responsible Scaling Policy was first published on 19 September 2023 and now stands at version 3.4, effective 8 July 2026 — a document the company wrote, interprets, and has revised repeatedly. Version 3.2 gave the Long-Term Benefit Trust power to request external review of risk reports and to approve the reviewer, a real transfer of authority and still an internal one. Self-binding of this kind is a promise, not a constraint.

The essay's achievement is to have made a first step specific enough that failure would be visible: either an outside team gets badges and publishes something Anthropic dislikes, soon enough to matter, or it does not. What it cannot supply is what all three steps presuppose. Somebody has to be entitled to set the pace and able to hold others to it, and no such body exists, in domestic or international law, for frontier AI. Amodei has written a careful case for why an authority of that kind is needed. He stops just short of where it would come from, which is the question the whole argument turns on.