Who Aligns the Aligners? Brief Legal Thoughts on the “AI Safety” Fights to Come

Dario Amodei, the CEO of Anthropic, has published an essay – We Must Pace The Frontier – in which he writes: I have worked on AI for the last twelve years because I believe it could dramatically raise the quality of human life. I’ve written often about these incredible benefits: I believe that AI could cure most major diseases in the next 5–10 years, greatly accelerate economic growth rates, create a world of abundance and empowerment, and usher in a renaissance of democracy and freedom. But – and there is always a but – …like many technologies before it, AI brings risks, and because it is such a powerful technology, these risks are serious. Those risks include, according to some, the complete destruction of the human race. See, e.g., Eliezer Yudkowsky confidently asserting today that if we do not institute immediate global techno-communism, instituting draconian government control over speech and publication of a type never before seen in any Western society, we are all going to die: There are currently zero things more important than "don't die to AI" becoming a bipartisan project rather than a Democrat-polarized issue. If you have any political capital you can spend on this, do it now, I beg of you. Life or death.— Eliezer Yudkowsky ⏹ (@ESYudkowsky) September 13, 2026 There is no evidence that this will happen. Some proponents of regulation tell us that the only response is the most extreme response available: total state control. There is no evidence that this response is correct, either. Nobody knows the answer and history is no guide, save that apocalyptic predictions about new technologies have, to date, all been wrong. History does provide a great deal of guidance, however, about the use and misuse of government power. It tells us that the state is in fact likely the worst possible custodian for the most powerful publication and data analysis technologies. This notwithstanding, to address this risk, Amodei proposes …building AI at a balanced rate that aims to ensure its safety while still achieving its benefits and grappling with important geopolitical dilemmas. To wit, regulation. As my regular readers will be aware, I have been engaged, on behalf of my clients, in legal combat with Internet censors around the world, agencies who think that they have both the standing, the competence, and the right to tell American companies what software they can write and run, for the better part of 18 months. I feel now is an appropriate time to offer my preliminary thoughts. Amodei’s Proposal Anthropic is, as David Sacks correctly pointed out on X, free to slow down its research and development efforts into AI at any time, to any extent it wishes. Amodei proposes something else: that everyone slow down together, under supervision. While Amodei initially writes that the “slowdown” should be voluntary, the plan would be to progress to legal regulatory regimes – meaning, this proposal necessarily involves the use of coercive state power – which software developers would be expected to obey on a compulsory basis: The most effective method of pacing is via regulation that targets all US frontier AI companies, as that covers even those who are unwilling to cooperate voluntarily. (Emphasis added.) His plan has three elements. Embedded Evaluators The first is for “Embedded Evaluators” – …employee-like access to a team of embedded third-party evaluators (such as METR), whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes. There is already a robust industry of third-party “safety” overseers for Web 2.0 – what the House Judiciary Committee has described as the “censorship-industrial complex.” The track record of these entities from the last time around tells us how this arrangement plays out in practice. “Evaluate this.” One well-known private actor in this space was the Global Alliance for Responsible Media, or GARM. GARM, a commercial enterprise, described itself as “a voluntary cross-industry initiative created in 2019 to address digital safety.” Among other things, GARM provided a range of policy frameworks and guidelines, among them “the Brand Safety Floor and the Adjacency Standards Framework, which have supported brand owners in their independent development of their own bespoke, brand-specific safety frameworks to ensure that their advertising dollars do not inadvertently support illegal or harmful content that damages their brands.” Although GARM disbanded in 2024, according to the House Judiciary Committee, during its active period GARM worked with global regulators to pressure companies like Twitter, now X Corp., to wield “significant collective power” to coercively influence Twitter’s moderation decisions, including “silencing President Trump,” and to procure boycotts of the platform if the platform refused to obey. He adds: This is the key step for verifiability of any pacing commitments, and has precedent in the banking industry, which sometimes involves regulatory “supervisors” embedded along with employees. As it happens, the banking analogy is the exact argument leading “misinformation/disinformation” (read: pro-censorship) academics employ to justify the UK’s Online Safety Act and similar regimes; see e.g. Alan Jagolinzer from Cambridge University, who comes from the world of financial accounting and analogizes speech regulatory regimes to post-2008 financial reporting regulations. The problem, of course, is that financial accounting fraud is not a constitutional right; speech is. In America, software development absent the intent to commit or facilitate the commission of a crime is, as a general rule, protected expression. I fail to see how standing up a new crop of NGOs to perform substantially the same function as the “Online Safety” NGOs, using the same methods – only, this time with NGO commissars possessing highly sensitive employee-like access to internal systems – will lead to a different or better result than it has so far. Democratic Coordination (aka Regulation) The second is “Democratic Coordination,” whereby Frontier AI companies within democratic countries coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress. Some forms of coordination that would be impactful for pacing are legally challenging, and will require government support. There are two aspects to this: (a) competition law and (b) content regulation law. From a competition law standpoint, the problem Anthropic has is simple. Anthropic and OpenAI are the largest players in the AI market, by some distance, and coordinating their policies, procedures, and “standards” with each other risks classification as an unlawful cartel. This would particularly be the case if, for example, the two giants aligned on pricing or terms – say, by conforming their API terms so that anyone who used a model that defected from the standards in the global marketplace (e.g., Kimi, Deepseek) would be ineligible to interact with OpenAI’s or Anthropic’s software. “Government support” for “legally challenging” coordination is a polite way of asking for an antitrust exemption to allow greater coordination between competitors in the name of “safety.” It is a problem any industry consortium of any type needs to account for and this would be no exception. From a content regulation standpoint, the language “common safety standards as well as limits on the rate of unchecked AI progress” paints a broad brush. This suggests that Anthropic envisages that practically any industry using its software – from manufacturing, to biotechnology, to news publication and copywriting – will require (a) de novo “safety” standards in relation to non-expressive conduct, and (b) limits on how quickly AI software itself can be developed. In foreign countries, particularly the United Kingdom and Europe, where national governments have fewer constitutional guardrails on their power, I would expect that both (a) and (b) can be legislated without much difficulty in legal terms. If the ease with which rules like the Online Safety Act and Digital Services Act were implemented is any indication, there should not be terribly much difficulty in political terms, either. The primary legal problem with this aspect of Anthropic’s proposal is in the United States, particularly with (b) – limits on how quickly software itself can be developed. Software development, software publication, and web hosting are inherently expressive activities. See. e.g., the Bernstein v. United States line of cases, as well as Smith v. California, Cubby v. CompuServe, the fact pattern of Stratton Oakmont v. Prodigy, and the related legislative history around 47 U.S.C. § 230. The differences between the United States and its allies on Web 2.0 date back to our very founding, and in both subsequent caselaw and subsequent statutes, America has chosen to protect that activity from state interference. To the extent Anthropic and its fellow-travelers intend for this aspect of their plan to restrict American citizens from either (a) developing AI models or (b) using models and published FOSS model weights from China, First Amendment issues are immediately apparent and the weight of the precedent militates against government regulation. Global Coordination The third limb of Amodei’s plan is “Global Coordination,” whereby [t]he US and other democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance. Authoritarian governments aren’t the only problem America has here. We live in an era in which most of the Western world, with the exception perhaps of the United States, has enacted comprehensive technology regulation statutes focused on yesterday’s tech: chiefly, search and social media.