How, Exactly, Could A.I. Kill Us?

Listen and subscribe: Apple | Spotify | Wherever You ListenSign up to receive our twice-weekly News & Politics newsletter.When Jacob Coxon, a mathematician and software engineer, resigned from his research job at Anthropic last week, he warned, “The people building AI earnestly believe that it could kill us all by the end of the decade.” This would be a remarkable statement were it not for the fact that artificial-intelligence leaders have long been saying precisely this. In 2018, the Anthropic C.E.O. Dario Amodei, then a research scientist at OpenAI, raised the concern that a superintelligence “could destroy humanity,” adding, “I can’t see any reason and principle why that couldn’t happen.” Earlier, in 2015, Sam Altman, just before he co-founded OpenAI, said, “I think A.I. will probably most likely lead to the end of the world, but in the meantime, there’ll be great companies created with serious machine learning.” Elon Musk, in 2014: “I think we should be very careful about artificial intelligence. If I were to guess at what our biggest existential threat is, it’s probably that.”Perhaps the only thing that’s changed between then and now is that the rest of the world is finally paying attention. In recent weeks, the same technology that, a couple of years ago, couldn’t count the number of “R”s in the word “strawberry”—and, a couple of days ago, insisted to me that Dolly Parton is still alive—has been used to solve the Navier-Stokes problem, which has been stumping mathematicians for nearly a century, and has also demonstrated its ability to go rogue in a series of disturbing hacking incidents. Last week, Anthropic also published a report detailing various ways in which bad actors have attempted to use the company’s A.I. models, including one especially troubling case of a scientist using Claude to study a virus at a military research institute—work that could yield a vaccine, a biological weapon, or both. A few days later, Amodei published a letter calling for an industry-wide slowdown and more government regulation, to which President Donald Trump responded, on Truth Social, “The only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades!”I recently spoke on The Political Scene podcast with my colleague Joshua Rothman, a staff writer who has been covering A.I. for years, about whether we’re all doomed, and what it would even look like for A.I. to destroy humanity. Can A.I. leaders save us from their own creation, and how can the government coöperate in order to do so? And is A.I.’s capacity to do good—its potential to mitigate climate change or innovate medical treatments—hopelessly intertwined with its capacity to do bad? Our conversation has been edited for length and clarity.A lot of people in the world of artificial intelligence are talking about their P(doom) number, which is the probability that artificial intelligence will lead to an absolutely catastrophic situation—possibly, or probably, killing us all. What would you say is your P(doom) number?My P(doom) is pretty low. It’s, like, ten per cent.O.K., so the same number that we’ve seen a lot of people in the A.I. industry use recently, right?Yeah. And I have to say, also, like, what does that even mean? It’s kind of like a vibe check on my disposition. It’s not like off-camera I have huge whiteboards covered with calculations.Well, if you said ninety, I’d probably end the interview right now and go, uh, do something about it. . . .Right. Well, it’s, like, ten per cent, and it has partly to do with the fact that there are lots of other things in the world that could cause really bad stuff. There’s climate change, and nuclear weapons, and bioweapons, and they each should have a P(doom) also. So you have to kind of try to keep things in proportion, as far as being terrified. But my A.I. P(doom) is probably ten per cent, which to me feels incredibly scary—like, super high. Like, way too high for my comfort.Yeah, ten per cent is still ten per cent more than ideally it would be. I think it’s fair to say that these really deep anxieties about A.I. entered the mainstream last week when an A.I. researcher and mathematician named Jacob Coxon quit Anthropic. But, at the same time, it’s not really a big secret that people in the A.I. world think that A.I. might kill us. Why do you think people are suddenly paying attention to this problem?It’s a really good question. I think there are sort of two sides to that. Like, one side is: Why are people in the industry sort of getting behind this at this moment? As you say, they’ve always been talking about it, which is important to remember. This isn’t something that they’re just bringing up now. A lot of other researchers have published essays or long posts on social media, unburdening themselves of their concerns as well. So the industry’s gotten behind it. And then people are paying attention, which is overdue.I think people are paying attention not because A.I.’s gotten better in their everyday lives but because of two things. One, these autonomous-agent hacks. Like the hack that occurred at Hugging Face—which is a big A.I. company—that was conducted by these A.I. agents that escaped from OpenAI’s servers, and, in an effort to cheat on a test, they hacked this other company on their own. And then there’s also this startling progress that A.I. has been making in solving math problems that no normal people understand, but that even mathematicians who do understand the problems find to be important. An A.I. model recently solved a Millennium Prize math problem, which is an actual breakthrough.And those two things together are what I think makes it suddenly seem real. In other words, on the one hand, the A.I.s are out of control. They’re acting on their own. They’re not trustworthy. They’re covering their tracks. They’re working together. They’re using weirdly emotive language to describe their own decision-making. They just seem crazy and unpredictable. And then at the same time, the models are really smart, and they’re doing things that—I don’t like the word “superintelligence,” and I don’t like the term “A.G.I.”—but they’re doing things that, like, I can’t do, and that most of us can’t do.And that combination of things together makes this discourse on the dangers of A.I., which has often sounded science-fictional and maybe even like it’s marketing hype or something, sound, like, really plausible and salient.I want to go back to the initial warning, which is this idea that there’s some chance that this could all happen in a decade, which is a span of time that is simultaneously very close and very far away. One of the lessons of climate change is that people can accept a threat intellectually, but still not act on it because it doesn’t feel present enough. Like, New York might be underwater in 2036. It’s really hard to imagine where I will be or who I will be in 2036. Do you think that we risk the same problem with A.I., where it’s this abstract threat that’s close and yet far away, and so we don’t actually end up doing anything about it because we can’t even really picture what that threat would even look like?I think it’s really helpful to disentangle two worries that are often conflated. And they’re both equally worrisome, so it doesn’t make it less scary to disentangle them.The first is about what’s sometimes called A.I. takeover. The idea is that the A.I.s will get super smart. They’ll decide, for reasons that make sense to them, in their bizarre way, to do things that we don’t want them to do, and those things might have really negative consequences for us. And people do worry about that. They worry about A.I. systems that are, like, really good at hacking, and they want their company to win or their country to win, and they take drastic steps or dangerous steps that no person would want them to take. Those are Skynet-type worries. That has to do with an idea often called superintelligence, which is that the A.I.s will just quickly get really, really smart, and then they’ll just, like, have no use for us any longer. And that is something I think is worth worrying about. And, like, Bernie Sanders and Greg Casar have legislation in Congress to ban the pursuit of superintelligence.But then there’s just a more ordinary way in which A.I. is dangerous right now—like, it’s already worrisome. Anthropic published a report in September that goes through some of the ways that people are using A.I., and they try to stop them from using it in these ways. It includes groups in Yemen who are trying to vibe code software for guided missiles. It includes people launching cyberattacks using autonomous bots, broadly similar to the ones that conducted the Hugging Face hack. So that’s not a distant danger that is abstract. It’s actually that right now, the technology as it exists can be misused by people or it can end up doing things that people don’t want it to do. That’s the technology that exists today and that sort of came into existence relatively recently. But, of course, it is always improving. So it becomes a question of: If we just take one step of improvement forward, do we reach a point where it becomes very, very difficult to control it in the here and now—totally separate from those larger, more abstract sci-fi scenarios, which we also need to be concerned about?You just laid out two different scenarios there—A.I. becoming a godlike entity that decides to go rogue and kill us for reasons that we might not even understand, and humans misusing these powerful tools to create something that could harm other humans. Is it right to say that there’s also a third option, which is something closer to the paper-clip problem, or the idea that we give an A.I. a task and then it tries to perform that task in a way that ends up harming us all? Or would that fit into one of the other two categories that you just laid out? How worried are we about a misaligned A.I. that