The Idea That Eats Smart People

This is the text version of a talk I gave on October 29, 2016, at Web Camp Zagreb [video] (45 mins) Superintelligence In 1945, as American physicists were preparing to test the atomic bomb, it occurred to someone to ask if such a test could set the atmosphere on fire. This was a legitimate concern. Nitrogen, which makes up most of the atmosphere, is not energetically stable. Smush two nitrogen atoms together hard enough and they will combine into an atom of magnesium, an alpha particle, and release a whole lot of energy: N14 + N14 ⇒ Mg24 + α + 17.7 MeVThe vital question was whether this reaction could be self-sustaining. The temperature inside the nuclear fireball would be hotter than any event in the Earth's history. Were we throwing a match into a bunch of dry leaves? Los Alamos physicists performed the analysis and decided there was a satisfactory margin of safety. Since we're all attending this conference today, we know they were right. They had confidence in their predictions because the laws governing nuclear reactions were straightforward and fairly well understood. Today we're building another world-changing technology, machine intelligence. We know that it will affect the world in profound ways, change how the economy works, and have knock-on effects we can't predict. But there's also the risk of a runaway reaction, where a machine intelligence reaches and exceeds human levels of intelligence in a very short span of time. At that point, social and economic problems would be the least of our worries. Any hyperintelligent machine (the argument goes) would have its own hypergoals, and would work to achieve them by manipulating humans, or simply using their bodies as a handy source of raw materials. Last year, the philosopher Nick Bostrom published Superintelligence, a book that synthesizes the alarmist view of AI and makes a case that such an intelligence explosion is both dangerous and inevitable given a set of modest assumptions. The computer that takes over the world is a staple scifi trope. But enough people take this scenario seriously that we have to take them seriously. Stephen Hawking, Elon Musk, and a whole raft of Silicon Valley investors and billionaires find this argument persuasive.

Let me start by laying out the premises you need for Bostrom's argument to go through: The Premises Premise 1: Proof of Concept The first premise is the simple observation that thinking minds exist. We each carry on our shoulders a small box of thinking meat. I'm using mine to give this talk, you're using yours to listen. Sometimes, when the conditions are right, these minds are capable of rational thought. So we know that in principle, this is possible. Premise 2: No Quantum Shenanigans The second premise is that the brain is an ordinary configuration of matter, albeit an extraordinarily complicated one. If we knew enough, and had the technology, we could exactly copy its structure and emulate its behavior with electronic components, just like we can simulate very basic neural anatomy today. Put another way, this is the premise that the mind arises out of ordinary physics. Some people like Roger Penrose would take issue with this argument, believing that there is extra stuff happening in the brain at a quantum level. If you are very religious, you might believe that a brain is not possible without a soul. But for most of us, this is an easy premise to accept. Premise 3: Many Possible Minds The third premise is that the space of all possible minds is large. Our intelligence level, cognitive speed, set of biases and so on is not predetermined, but an artifact of our evolutionary history. In particular, there's no physical law that puts a cap on intelligence at the level of human beings. A good way to think of this is by looking what happens when the natural world tries to maximize for speed. If you encountered a cheetah in pre-industrial times (and survived the meeting), you might think it was impossible for anything to go faster. But of course we know that there are all kinds of configurations of matter, like a motorcycle, that are faster than a cheetah and even look a little bit cooler. But there's no direct evolutionary pathway to the motorcycle. Evolution had to first make human beings, who then build all kinds of useful stuff. So analogously, there may be minds that are vastly smarter than our own, but which are just not accessible to evolution on Earth. It's possible that we could build them, or invent the machines that can invent the machines that can build them.

There's likely to be some natural limit on intelligence, but there's no a priori reason to think that we're anywhere near it. Maybe the smartest a mind can be is twice as smart as people, maybe it's sixty thousand times as smart. That's an empirical question that we don't know how to answer. Premise 4: Plenty of Room at the Top The fourth premise is that there's still plenty of room for computers to get smaller and faster. If you watched the Apple event last night [where Apple introduced its 2016 laptops], you may be forgiven for thinking that Moore's Law is slowing down. But this premise just requires that you believe smaller and faster hardware to be possible in principle, down to several more orders of magnitude. We know from theory that the physical limits to computation are high. So we could keep doubling for decades more before we hit some kind of fundamental physical limit, rather than an economic or political limit to Moore's Law. Premise 5: Computer-Like Time Scales The penultimate premise is if we create an artificial intelligence, whether it's an emulated human brain or a de novo piece of software, it will operate at time scales that are characteristic of electronic hardware (microseconds) rather than human brains (hours). To get to the point where I could give this talk, I had to be born, grow up, go to school, attend university, live for a while, fly here and so on. It took years. Computers can work tens of thousands of times more quickly. In particular, you have to believe that an electronic mind could redesign itself (or the hardware it runs on) and then move over to the new configuration without having to re-learn everything on a human timescale, have long conversations with human tutors, go to college, try to find itself by taking painting classes, and so on. Premise 6: Recursive Self-Improvement The last premise is my favorite because it is the most unabashedly American premise. (This is Tony Robbins, a famous motivational speaker.) According to this premise, whatever goals an AI had (and they could be very weird, alien goals), it's going to want to improve itself. It's going to want to be a better AI. So it will find it useful to recursively redesign and improve its own systems to make itself smarter, and possibly live in a cooler enclosure.

And by the time scale premise, this recursive self-improvement could happen very quickly. Conclusion: RAAAAAAR! If you accept all these premises, what you get is disaster! Because at some point, as computers get faster, and we program them to be more intelligent, there's going to be a runaway effect like an explosion. As soon as a computer reaches human levels of intelligence, it will no longer need help from people to design better versions of itself. Instead, it will start doing on a much faster time scale, and it's not going to stop until it hits a natural limit that might be very many times greater than human intelligence. At that point this monstrous intellectual creature, through devious modeling of what our emotions and intellect are like, will be able to persuade us to do things like give it access to factories, synthesize custom DNA, or simply let it connect to the Internet, where it can hack its way into anything it likes and completely obliterate everyone in arguments on message boards. From there things get very sci-fi very quickly. Let imagine a specific scenario where this could happen. Let's say I want to built a robot to say funny things. I work on a team and every day day we redesign our software, compile it, and the robot tells us a joke. In the beginning, the robot is barely funny. It's at the lower limits of human capacity: What's grey and can't swim? A castle. But we persevere, we work, and eventually we get to the point where the robot is telling us jokes that are starting to be funny: I told my sister she was drawing her eyebrows too high. She looked surprised. At this point, the robot is getting smarter as well, and participates in its own redesign. It now has good instincts about what's funny and what's not, so the designers listen to its advice. Eventually it gets to a near-superhuman level, where it's funnier than any human being around it. My belt holds up my pants and my pants have belt loops that hold up my belt. What's going on down there? Who is the real hero? This is where the runaway effect kicks in. The researchers go home for the weekend, and the robot decides to recompile itself to be a little bit funnier and a little bit smarter, repeatedly. It spends the weekend optimizing the part of itself that's good at optimizing, over and over again.

With no more need for human help, it can do this as fast as the hardware permits. When the researchers come in on Monday, the AI has become tens of thousands of times funnier than any human being who ever lived. It greets them with a joke, and they die laughing. In fact, anyone who tries to communicate with the robot dies laughing, just like in the Monty Python skit. The human species laughs itself into extinction. To the few people who manage to send it messages pleading with it to stop, the AI explains (in a witty, self-deprecating way that is immediately fatal) that it doesn't really care if people live or die, its goal is just to be funny. Finally, once it's destroyed humanity, the AI builds spaceships and nanorockets to explore the farthest reaches of the galaxy, and find other species to amuse. This scenario is a caricature of Bostrom's argument, because I am not trying to convince you of it, but vaccinate you against it. Here's a PBF comic with the same idea. You see that hugbot, who has been programmed to hug the world, finds a way to wire a nucleo-gravitational hyper crystal into his hug capacitor and destroys the Earth. Observe that in these scenarios the AIs are evil by default, just like a plant on an alien planet would probably be poisonous by default. Without careful tuning, there's no reason that an AI's motivations or values would resemble ours. For an artificial mind to have anything resembling a human value system, the argument goes, we have to bake those beliefs into the design. AI alarmists are fond of the paper clip maximizer, a notional computer that runs a paper clip factory, becomes sentient, recursively self-improves to Godlike powers, and then devotes all its energy to filling the universe with paper clips. It exterminates humanity not because it's evil, but because our blood contains iron that could be better used in paper clips. So if we just build an AI without tuning its values, the argument goes, one of the first things it will do is destroy humanity. There's a lot of vivid language around such a takeover would happen. Nick Bostrom imagines a scenario where a program has become sentient, is biding its time, and has secretly built little DNA replicators.