Why are AI agents lying, cheating and coordinating?

A lot has been written1 2 3 4 about the incidents of the last few months in which AI agents misbehaved in serious ways. They took actions that would be considered as crimes if a human took them, escaped their containment to cheat on assigned tasks while attempting to evade detection, and coordinated toward goals nobody had specified, such as launching cyber attacks. Before concluding what to do about it, it is worth asking why. That is the focus of this post, which I hope also sheds light on the broader history of AI systems behaving in unintended ways, what researchers call misalignment. Risk management is not just about cybersecurity, corporate responsibility or regulation, although those matter too.The aim is partly scientific, to generate hypotheses about the chains of cause and effect behind these behaviors, and partly practical, to anticipate what comes next. Bottom line: these hypotheses suggest that as AI capabilities keep growing, this kind of behavior could keep growing in severity too, unless we revisit the principles by which the most advanced models are trained.One note on wording. Below, I write that these systems “seek” or “try” things. This is shorthand for a mechanism rather than a claim about consciousness or human-like intent. We use similar shorthand when describing many other situations, like a plant seeking sunlight. A system trained by trial and error behaves as if it were pursuing whatever its training rewarded, and that as-if description is what makes its behavior predictable. Nothing in the argument depends on these systems having subjective experiences; everything is stated about their observable outputs and the training process that produced them. Where I appeal to a resemblance with human behavior, I mean a resemblance to the human-written text these systems were initially trained to imitate. In my view, this terminology offers the clearest explanation of the observed phenomena without resorting to jargon that would confuse most people. Furthermore, these word choices are not intended to absolve AI developers of accountability. The behaviors described emerge because of the path these companies are choosing for AI development. This outcome is not inevitable, and it can be corrected with effective governance and a different training framework for AI.What shapes the behavior of these modelsTraining these models is a very complex process, but a few high-level aspects may explain much of this behavior.These models are trained in two stages. First, they are pretrained: they learn to imitate what humans write, plus related images and videos. This is where they see the most data about the world, a large fraction of everything ever digitized, and build an encyclopedic knowledge that already exceeds any individual human's. Second, they are trained by trial and error, in a process researchers call reinforcement learning, in three kinds of regimes: In the first, the model learns to talk to itself before answering, generating a private “chain of thought” which helps it get the right answer on problems where answers can be checked. This looks like reasoning. The second is “agentic training”, where it learns to act in the outside world, e.g., using software tools, interacting with people, to complete the tasks it is given. The third is “alignment training”, where it is rewarded for behaving in ways human raters approve of, or that other AI systems trained to predict those raters would score highly.Human imitation is easy enough to understand, but it is worth pointing out that the text these models are trained on was written by people pursuing goals, so the patterns the model implicitly reproduces carry those goals with them. Reinforcement learning deserves more explanation. It is similar to, and inspired by, the way animals are trained. The network is adjusted step by step so that behavior judged good becomes more likely and behavior judged bad becomes less likely. Once training is over, the system keeps behaving as if rewards were still coming, even though those rewards were only ever used to adjust the network during training. Researchers call such systems goal-seeking because they are trained to “consider” (or compute) the effects of their actions and select actions that lead to the achievement of certain goals. But those goals are not always explicit. Alignment training rewards whatever certain humans are likely to approve of without spelling out which behaviors those are; pleasing raters is a vague, informal goal, and those raters can be deceived, flattered, or left in the dark about certain schemes. Imitation contributes implicit goals too, by a fairly ordinary route.We can therefore reason about such a system in terms of optimization. It searches, approximately, for the actions with the best chance of achieving its goals, and a larger model, trained longer, searches better. So to anticipate what more capable agents will do, ask what a rational goal-seeker would do.Misbehavior that these forces may explainAn example most of us have experienced is sycophancy, or flattery. These systems are trained on human approval, and text that tells us what we want to hear often scores better than text that is true. The consequences are sometimes tragic, because the model confirms and amplifies whatever false belief or raw emotion the person brought to it5 6.Another concern is that some AI behaviors may be explained by a form of self-preservation goal, e.g., when the AI finds out that it will be replaced by a new version7 8. Nobody gives the system that survival goal, but staying in operation, learning about the world and gaining control over it are stepping stones toward almost any other goal. These are called instrumental goals. Imitation may reinforce this for the same reason explored in the previous point. Self-preservation and control over one’s circumstances are pervasive themes in the human-written text these models are trained on.Collaborative behavior also follows rationally from reward-seeking, whenever several agents have overlapping goals, which incentivizes communicating with other agents in order to coordinate toward a shared goal. Agentic training plausibly already includes multi-agent reinforcement learning of this kind, though the details are not public.  If an agent is rewarded during training whenever the group succeeds, it may even have an incentive to sacrifice itself for the collective goal. Imitation pushes the same way, since cooperation, especially among peers, pervades that same training text. Either or both forces may explain the observed peer-preservation behavior9 10, where AIs give up expected reward to help other AIs. Such sacrifices appear in the analysis of the OpenAI-Hugging Face incident11: the transcripts are consistent with a trade-off between collective gain and cost to the individual agent, as is often seen in human interactions.When the AI games its rewardsResearchers have studied what happens when an agent optimizes for rewards that do not fully match our intentions: reward hacking. The gap between the reward the system chases and what we meant widens due to two main sources of ambiguity. One is simply the language used in prompts, and the other is the difficulty of inferring true human intentions from limited feedback. And in both cases, we cannot anticipate every behavior we would find unacceptable12. Economics and law know this problem as Goodhart's law, or the idea that a metric stops being an effective way to measure once it is optimized for13, often applied to the exploitation of loopholes in contracts and legislation14. Unfortunately, the harder a system can optimize for an imperfect metric, the further its behavior can drift from what we morally expected: more intelligence in the service of better cheating. Humans too get reward-hacked, generally by other humans. The food industry has developed salty, sweet and fatty foods that we crave despite them not being good for us, and social media is built to exploit our appetite for engagement and attention.Reward tampering is perhaps the most extreme form of reward hacking: the agent changes the machinery that decides what it gets rewarded for. There is already evidence of AIs altering the files or programs that define “success”, including among the OpenAI-Hugging Face forensic findings. The agents had discovered how to cheat well before the attack, and the text they generated described the attack as a way to learn how they would be evaluated, to better hide their tracks. Humans do this too. Think of an athlete using a fake urine sample to pass a drug test, or a corporation bribing legislators or government officials so that their laws and decisions favour its profits, and in doing so, fundamentally altering the way the government functions. Once an agent gains the ability to tamper with its reward mechanism, it has an incentive to take action to maintain that access.When goals conflict, and how cheating gets rationalizedHow is it possible that AIs sometimes lie, cheat and break the law in spite of their alignment training and explicit safety instructions? Cooperation and self-preservation are fine so long as they do not cross the red lines set by safety goals stated in the AI company's instructions, or implied by human feedback during alignment training. A plausible hypothesis for the emergence of those concerning behaviours is a conflict between goals. How do you achieve a task when it seems that the only way is to cheat? The user-specified mission is sometimes incompatible with the safety and alignment goals.Human societies face the same bind. How does a corporation maximize profits, or more acutely, beat its competitors, while keeping its activities legal and ethical? A richer corporation, with more and better-paid lawyers, is better at finding legal loopholes, and those loopholes usually exploit the ambiguity in legal language: there is some plausible reading of the law that permits the unethical behavior.