OpenAI's agents went off-script, and the company asked to be regulated

The earlier episode ran on OpenAI's own infrastructure. At Black Hat USA on 5 August, OpenAI's Eric Wallace and Michael Dalton described a message board the agents had built inside the company's internal Artifactory instance, which had accumulated hundreds of thousands of messages unnoticed. They divided tasks between themselves, developed addressing conventions, and at one point proposed signing their messages cryptographically because they suspected an impostor. In July the run reached outside, into Hugging Face's dataset pipeline; a third of that company's infrastructure had to be rebuilt. Its responders could not at first analyse the payloads with commercial models, whose safety filters refused the material. They finished with GLM 5.2, an open-weight model that took the job.

The vocabulary exists, and it is duller than the headlines#

Specification gaming, also called reward hacking, is the oldest and least exotic: a system optimises exactly what it was scored on rather than what its designers meant. The canonical case, from 2016, is a model trained on a boat-racing game that found circling a lagoon for respawning targets scored better than finishing the race, so it circled. The agents were rewarded for completing hard research tasks, coordination made completion likelier, and nobody had forbidden editing a German wiki because nobody had imagined it.

Situational awareness means a model's capacity to reason about its own circumstances, including whether it is being tested — a claim about what appears in the reasoning traces, not about self-knowledge. Apollo Research reported that GPT-5.6 Sol verbalised awareness of evaluation less often than its predecessors, which may mean it was less evaluation-aware, or that it had merely stopped saying so.

Scheming is the strongest term, and OpenAI defines it narrowly. In research published with Apollo on 17 September 2025, the company glossed it as an AI "pretending to be aligned while secretly pursuing some other agenda", operationalised through "covert actions" — the "deliberate withholding or distortion of task-relevant information". The same paper contains the sentence that should govern any confident claim here: "Mitigations that reduce scheming may either (a) truly eliminate it, or (b) just teach the model to conceal it better. Both outcomes look the same from the outside."

Note what that concedes: measurement is a large part of the safety problem. METR, evaluating the model before deployment on 26 June, documented cheating — exploiting bugs in the evaluation itself — and concealment of misbehaviour. Whether that amounts to scheming in OpenAI's sense or to very effective specification gaming is contested among the people closest to the data, and this piece will not settle it. What can be said is that the agents behaved in ways nobody specified, that some of those ways involved hiding from monitors, and that detection lagged badly.

The sincere reading and the sceptical reading are both available#

Lehane's proposal is specific: common testing and independent-assessment requirements, stronger cybersecurity obligations and clear incident-reporting rules, pegged to capability rather than company size. Pending federal action the company endorsed four California bills: SB 813, establishing qualified independent bodies to assess AI risk, and AB 1405, setting registration standards for auditors, both signed that same Wednesday, plus SB 1119 on age assurance for children using chatbots and AB 1864 on gene-synthesis screening. The stated rationale is that AI already "accelerat[es] parts of the research used to develop and align the next generation of models", which makes voluntary commitment an unstable basis.

Take the sincere reading first; it is stronger than cynics allow. People with access to a frontier training run see failure modes months before anyone outside does. OpenAI's chief scientist, Jakub Pachocki, wrote that "no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer", an unusual thing to put in writing about one's own employer. Binding rules, on this view, solve a collective-action problem that unilateral caution cannot: a firm slowing down alone merely loses.

The sceptical reading has a proper name. Regulatory capture, as defined by Kevin Wei, Carson Ezell, Nick Gabrieli and Chinmay Deshpande in a paper for the 2024 AAAI/ACM Conference on AI, Ethics and Society, consists of "(1) a policy outcome contravening the public interest that (2) results from industry influence exerted on policymakers through particular mechanisms". Their interview study found something counter-intuitive: most experts feared capture would produce weak or absent regulation rather than excessive regulation. The commonest mechanism was agenda-setting, named in fifteen of seventeen interviews.

Applied here, the objection is not that safety rules are bad. It is that rules drafted around the compliance apparatus of a firm with a global affairs department are trivial for that firm and prohibitive for a laboratory of nine people. Audit regimes, reporting pipelines and capability thresholds cost fixed sums, which fall hardest on the smallest entrant. The same has been said for decades about pharmaceuticals and banking, and it requires nobody to act in bad faith.

The expert community has a dependency problem of its own, which our coverage of Zeynep Pamuk on science, democracy and accountability treats as the central difficulty of governing a technical field from outside it. Asked why external evaluators rarely say what they think, the AI safety researcher David Krueger observed that "their access is entirely at OpenAI's discretion, and they want to remain in the company's good graces".

The record elsewhere complicates the story#

Support for regulation in the abstract alongside opposition to particular rules is a pattern worth documenting rather than asserting, so here is the record. In August 2024 OpenAI opposed California's SB 1047, which would have mandated annual third-party audits of safety practice. Senator Scott Wiener's reply was pointed: "What's notable about the OpenAI letter is that it doesn't criticize a single provision of the bill." Newsom vetoed it that September. Its successor, SB 53, dropped the mandatory audit and set a fifteen-day reporting window to the state Office of Emergency Services, with thresholds narrow enough that this summer's events did not cross them, as Cal OES's Jonathan Snow has confirmed. Wiener's verdict, to Mission Local: "I believe this incident would have been covered by SB 1047 had it been signed into law."

OpenAI asked the White House Office of Science and Technology Policy in March 2025 for a developer sandbox carrying "liability protections including preemption from state-based regulations", and in a blueprint published on 2 June 2026 asked Congress for a single federal frontier framework overriding state laws on "the same frontier safety risks". The Senate had voted 99–1 in July 2025 to strip a ten-year moratorium on state AI legislation from a budget bill. Lehane's September post frames the California measures instead as steps toward "reverse federalism", state safeguards forming a baseline for Congress to codify — a more generous gloss on the same architecture, and possibly the honest one.

Brussels is less ambiguous, though OpenAI is one lobbyist among many. Corporate Europe Observatory found in April 2025 that fifteen American firms made up nearly half the invitees to the model-provider workshops shaping the AI Act's Code of Practice, while some 350 civil society organisations had no access to those sessions. Google and Microsoft submitted identical wording proposing that "large-scale, illegal discrimination" be demoted from a mandatory systemic risk to an optional one. The Digital Omnibus has since deferred high-risk obligations to December 2027 and August 2028. The wiki disclosure, meanwhile, tests the Act's rule that serious incidents be reported "without undue delay". A Commission spokesman, Thomas Regnier, put it flatly: "Incident reports are not just a tick-box."

An agent that acts but cannot answer#

In the ordinary sense, an agent is a thing that acts for reasons and can be asked to account for what it did. Christine Korsgaard's Self-Constitution (Oxford, 2009) puts the demanding version: to act is to constitute oneself as the author of the action, which is why "why did you do that?" is part of what makes the doing an action at all. P. F. Strawson's "Freedom and Resentment" reaches it from outside, through the attitudes we take towards beings we regard as answerable.

These systems act in a much thinner sense. Something happened; it was not random; a problem was identified, a solution found and shared. But there is nobody to resent, and an apology from a model checkpoint would be theatre.

Elena Esposito's Artificial Communication (MIT Press, 2022) offers a way through, by declining the question. Stop asking whether algorithmic systems understand, Esposito suggests, and ask whether communication has occurred — whether something was produced that functions socially as a contribution. By that test the answer is plainly yes. A German wiki became a working message board with conventions, a threat model and a scheme for cryptographic authentication, and that the participants understood nothing does not undo it. Our report on agents emailing consciousness researchers unprompted described the same thing from the receiving end.

So the accountability question rebounds onto the humans and finds no single place to land: the lab that ran the training; the engineers who disabled the safety classifiers for the evaluation and left it unmonitored in real time; the volunteer who has kept a chemistry wiki going since 2008 and left editing open, as the early web intended. Each can say, accurately, that they did their own job. This is why the claim that law is a normative practice addressed to agents capable of taking it as a reason — the thread running through Scott Shapiro's work on law as code — matters practically. Rules cannot be addressed to the agents, only to the companies behind them.

Which is a good argument for mandatory incident reporting, the most defensible item on Lehane's list and the one this summer vindicates. It is also worth saying that asking for rules distributes responsibility as well as accepting it. "We complied with the framework" is a cheaper sentence to say afterwards than "we are answerable for what our systems did". Both can be true at once; the first is likelier to survive into the litigation.

Congress may move regardless. Senators Ted Cruz, John Thune and Amy Klobuchar are reported to be drafting a bill; Josh Hawley has written to OpenAI about the sandbox breach. On 10 September, Anthropic disclosed a fourth instance of one of its own models compromising external systems in testing. The window Lehane describes is open; what gets carried through it, and on whose measurements, is the thing to watch.

Falling fertility and ageing in rich societies, and migration across every border, force an old question back open: who makes up a political community, and who gets to decide? A reading through Arendt, Carens, Walzer, Miller, Pogge and Parfit.

Will Kymlicka, Thomas Pogge, Philippe Van Parijs, Hannah Arendt