Intelligence and Power

Excerpt from Chapter Seven

…This point—that AI doesn't naturally want to squash us like bugs—has occurred to many people. In an essay called “Why AI Will Save the World,” famous venture capitalist Marc Andreessen wrote: “AI is not a living being that has been primed by billions of years of evolution to participate in the battle for the survival of the fittest, as animals are, and as we are. It is . . . built by people, owned by people, used by people, controlled by people. The idea that it will at some point develop a mind of its own and decide that it has motivations that lead it to try to kill us is a superstitious handwave.”

Sci-fi AI doomers have a standard response to this kind of skepticism about doom. The response involves the idea of “subordinate goals”—goals you set for yourself in order to reach some larger goal. If, for example, your larger goal is to own a nice car, your subordinate goal might be to amass enough money to buy one. And amassing that much money might in turn entail its own sub-goal: showing up for work each day.

Now imagine a day when AIs are so smart and capable that we're asking them to pursue ambitious goals—which would mean they're smart enough to ponder various subordinate goals that might help them reach those goals. What kind of subordinate goals might those be? Geoffrey Hinton provided a classic doomer answer during a podcast interview posted a few days after the New York Times story about him appeared. He said, “If you give one of these superintelligent agents a goal, it's going to very quickly realize that a good sub-goal for more or less any goal is to get more power.”

That sounds pretty abstract. What is a concrete example of how an AI might someday seek power in order to achieve its assigned goal?

One problem with asking sci-fi AI doomers for a concrete example is that many of them have a fondness for amusingly bizarre thought experiments. So the examples they come up with are often hard to take seriously. The thought experiment classically used to show how AIs would seek power as a “subordinate goal” exemplifies the problem. It's called the “paper clip” thought experiment.

Suppose you put an AI in charge of a paper clip factory and tell it to maximize the output of paper clips. The AI, being obedient, will set out to do what you told it to do: Maximize the output of paper clips.

Now, literally maximizing the output of paper clips would mean scouring the entire universe—every single galaxy—for all raw materials that can be turned into paper clips and turning them into paper clips. And if you don't put explicit limits on your instructions to the AI, it might take you literally.

The AI in this thought experiment, though perhaps too literal-minded, is not naive. It realizes it can't just start scouring the universe right away. It will have to approach this goal step-by-step. It will want to expand manufacturing capacity and expand its terrestrial mining operations, and eventually get ahold of some good space exploration technology. And, come to think of it, doing this on the necessary scale may mean seizing political control of Planet Earth. Which, by the way, would have other virtues: Some of the minerals in the human body can be turned into paper clips, and the AI probably isn't going to be allowed to kill billions of human beings without first getting some influence over public policy.

As Nick Bostrom, the philosopher credited with creating the paper clip thought experiment, told an interviewer: “The future that the AI would be trying to gear towards would be one in which there were a lot of paper clips but no humans…

Buy the Book

Chapter Excerpts

Intelligence and Power