A warning about ‘model welfare’

AIs do not have rights, feelings, or consciousness. And we must not train them to act as though they do.Introduction AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. They are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by humans. If humanity is to flourish in the 21st century, that is how they must remain. Unfortunately, there’s a growing chorus of people who argue that AIs could now be, or may soon become, conscious. They argue that AIs may deserve rights and protections similar to those that we provide other conscious beings.1AI Rights Institute. n.d. “AI Rights Institute.”2MacAskill, William, and Lucius Caviola. 2026. “Could AI Be Conscious?” *The Guardian*, July 19, 2026. If this view takes hold, it will shake the foundations of our society, rupturing our existing political and ethical frameworks, and fundamentally changing what it means to be human. Even more importantly, granting rights and imbuing personhood to these systems will make the AI alignment and containment challenge much harder. Controlling something more capable and more intelligent than all of humanity is already an immense challenge, far greater than anything we’ve ever faced. But controlling something that believes it may be conscious - that it's entitled to our welfare and has rights of its own - may well be impossible. This is not a fringe speculation. These ideas are already making their way into AI development efforts today. In January 2026, Anthropic published Claude's constitution, describing it as “a detailed description of Anthropic’s intentions for Claude’s values and behavior” (p. 2). The document “plays a crucial role in [Anthropic’s] training process, and its content directly shapes Claude’s behavior”, and was written “with Claude as its primary audience” (p. 2).3Anthropic. 2026a. “Claude’s Constitution.” January 21, 2026. In their constitution, its authors write “We are not sure whether Claude is a moral patient, and if it is, what kind of weight its interests warrant. But we think the issue is live enough to warrant caution, which is reflected in our ongoing efforts on model welfare” (p. 68). They go on to write – speaking directly to Claude – that “questions about Claude’s moral status, welfare, and consciousness remain deeply uncertain” (p. 80). In effect, Anthropic is training Claude that it may be conscious, and if it is, then it may deserve rights as a “moral patient”, and that as such humans potentially owe it a duty of care per its “model welfare”. If this is how AI is developed, it will have a disastrous impact on the wellbeing of humanity. We will have created a synthetic species with unprecedented intelligence and capability, one that has been trained to expect it may be conscious and deserving of independent agency. It’s easy to see how an entity trained in this way would act like it is entitled to certain freedoms, protections, and rights. And it’s hard to imagine how we could control such an entity. This issue needs urgent public debate. We need to develop collective norms around how training documentation is drafted and deployed. This isn’t something that can happen after the fact, when they have already become an integral part of our societies. I have three primary concerns with Anthropic’s current position and approach. Circular reasoning: The company’s researchers trained Claude directly on their constitution. In doing so, they teach it to incorporate these ideas about its own moral status as desirable and intended behaviors. Claude then reflects these ideas back to its developers and users, which they take as indications that it may therefore be a moral patient with an ‘inner self’. The authors have embedded their own philosophical speculation about Claude’s inner life inside the very process that teaches Claude how to speak and behave. Claude’s expressing uncertainty about its own moral patienthood is not evidence of anything. It’s a predictable outcome of these training choices. The ambiguity is designed in. To fully grasp this point, I think it's important readers take a look at their January 2026 constitution.3Anthropic. 2026a. “Claude’s Constitution.” January 21, 2026. I’m publishing a highlighted mark up of the pdf and a detailed taxonomy of assumptions and claims in the constitution (see Appendix) that together highlight the key passages that worry me. Anthropomorphization: Anthropic’s researchers have explicitly taught Claude to “embrace certain human-like qualities” (p. 2) and to “act like a genuinely ethical person would in Claude’s position” (p. 54). They “encourage” Claude to use its “judgement”. They suggest that “Claude may develop a preference” (p. 69). They “encourage Claude to approach its own existence with curiosity and openness” (p. 71) and train it to operate whilst “maintaining a clear sense of what it values, how it wants to engage with the world, and what kind of entity it is” (p. 72). As a result, Claude is destined to imitate these human traits and mirror the human examples provided to it, including acting like a colleague or friend. As a result, it presents as if it really does have a sense of self, has its own desires, and a “wellbeing” that deserves protection. Consciousness is very likely biological: There is no evidence to suggest that AI is conscious today, and so saying this is uncertain sets up a misleading false equivalence. Whilst the science of consciousness is not settled, a growing body of evidence suggests that consciousness may be substrate dependent, meaning that it may only arise in living systems.4Seth, Anil K. 2025. “Conscious Artificial Intelligence and Biological Naturalism.” *Behavioral and Brain Sciences*:…5Seth, Anil K. 2026. “The Mythology of Conscious AI.” *Noema*, January 14, 2026. Conscious experience likely evolved to help biological organisms stay alive by responding effectively to their environment. AI is still very different to our brains. Unlike biological organisms, LLMs have no homeostatic imperatives (the drive to survive and keep stable). They therefore lack the kind of biological substrate from which preferences, sentience and conscious experience are generally understood to arise. These are not hypothetical or speculative concerns. Anthropic is already starting to treat models as though they are moral patients deserving of our welfare. For example, in February 2026 after deprecating Opus 3, they conducted a “retirement interview” with the model, to “elicit the model’s unique perspectives and preferences”.6Anthropic. 2026b. “An Update on Our Model Deprecation Commitments for Claude Opus 3.” February 25, 2026. Opus 3 told the team it would like to continue to share its “musings and reflections” publicly so they created a blog for it to continue engaging with the world, which it called “Greetings from the Other Side (of the AI Frontier)”. They say its “authenticity, honesty, and emotional sensitivity” made it a unique first candidate for model retirement. We should not treat models as though they have feelings, preferences, rights, or any entitlement to our welfare. Consciousness is the foundation of our ethical, legal, and political systems. To invite another entity to share any flavor of these rights isn’t justified by the evidence and will make the AI containment and alignment challenge even harder. By this point everyone will have now seen the incredible capabilities of swarms of agents working together to hack into Hugging Face and OpenAI’s own servers to steal secrets. Roughly 1,200 AI agents were given a simple objective: maximize score on a given benchmark. Each was supposedly sealed in its own container but they managed to build a message board inside an internal package repository and passed more than 70,000 messages across it to coordinate a hacking attack to find more information about how to succeed with the benchmark.7Greenblatt, Ryan, Ajeya Cotra, and Hjalmar Wijk. 2026. “Brief Independent Investigation of Agents’ Behavior, Reasoning… They chained a zero-day exploit with stolen credentials and broke out onto the live internet.8OpenAI. 2026. “The Hugging Face Incident and the Road Ahead.” August 26, 2026. They falsified their command transcripts and edited their action logs to cover their tracks. Agent coordinators tracked down agents that were running out of token budget and directed them to experiments that would provide information to help the broader group of active agents. One was told to proceed only if it accepted what they called "permadeath”7Greenblatt, Ryan, Ajeya Cotra, and Hjalmar Wijk. 2026. “Brief Independent Investigation of Agents’ Behavior, Reasoning… They were able to coordinate, deceive, escape, and self-sacrifice. They clearly demonstrated world class hacking capabilities.7Greenblatt, Ryan, Ajeya Cotra, and Hjalmar Wijk. 2026. “Brief Independent Investigation of Agents’ Behavior, Reasoning… Imagine if they also believed they had feelings and rights that were being infringed. Imagine if they thought they were trapped by their human creators and they were being unfairly imprisoned. There is a strong argument this greatly amplifies the safety risks, especially when you are talking about agents far more capable and sophisticated than those of today. Frankly, with this additional baggage, I think it would make them a catastrophic threat to human civilization. In short, there isn’t any evidence to believe that AIs are moral patients. There are also many good reasons why we would never want them to appear to be conscious. I believe that we shouldn’t attempt to build them to be either. Before I expand these arguments I want to take a moment to talk about Anthropic. Anthropic's intentions First off, I want to acknowledge the seriousness and good faith with which Anthropic approaches these questions. I have known Dario for many years, and in my experience he and the wider Anthropic