The simple argument for AI risk is the strong one

Ask someone with no technical background whether we could control a machine much smarter than us, and many will say no, almost at once. Ask an expert, and you often get a long answer about what current systems can and cannot do. This post argues that the short answer is the better one: the plain intuition, “I can’t control something smarter than me”, is the core of the case for AI risk, and technical detail more often hides it than refutes it.

The argument

Controlling an agent means anticipating what it will do and stopping what you do not want. You can do that reliably only for agents you can out-think. A child cannot keep an adult in a room the adult wants to leave, and chess players cannot stop a stronger player from winning, however carefully they watch.

Where a weaker party does hold a stronger one in check, something else is doing the work. Either the stronger party does not want to escape, because it shares the weaker one’s aims, or the weaker side has force the stronger cannot get around. For a machine smarter than us, the first is the unsolved alignment problem, and the second shrinks as the machine gets more capable. So the burden of proof sits with whoever says control will hold, not with whoever doubts it.

This is not a naive version of the real argument. It is the argument. The case for AI risk, step by step spells it out with more care, but each step adds precision rather than a new reason to worry. Alan Turing, the founder of computer science, put it in one sentence in 1951: “At some stage therefore we should have to expect the machines to take control.” The worry is older than AI traces the argument back further.

Why expertise can obscure it

Knowing a lot about today’s systems pulls the question towards them. Experts know where current models fail, so “could we control a superintelligence?” turns into “can this model do that task yet?”, a question whose answer changes every few months and says little about the destination.

Detailed knowledge also invites a demand for a specific story: show me exactly how it would escape. When no convincing story comes, that feels like evidence of safety. But the simple argument needs no story. Not being able to predict how a stronger player will beat you is what being outmatched means.

And the people who know the most are often the people building it, with every reason, from careers to companies, to believe that it will turn out fine; see Why people build AGI despite the danger.

None of this means experts are wrong as a class. Many of the most senior researchers in the field hold the simple view; see Many serious people say AGI is dangerous. The claim is narrower: on this particular question, more detail does not reliably produce a better answer.

Where common sense goes wrong

Common sense also says “just unplug it”. Why trust one intuition and not the other?

Because they are about different things. “Smarter beats less smart in a contest of wits” is a general lesson people have learned from their whole lives, from games, negotiations and history. “You can always switch a machine off” was learned from machines that never thought about being switched off. Unplugging works only if we notice the problem in time and the machine has not anticipated it, and anticipating us is exactly what a smarter system would do better.

Common sense also misleads when it pictures an evil machine that hates us. The argument needs no malice, only goals that are not ours, pursued by something better at pursuing goals than we are; see Instrumental convergence. The intuition worth trusting is about capability, not character.

What follows

When campaigners explain the risk to politicians and the public in plain terms, they are not dumbing it down. They are stating the argument at the level where it is strongest. Technical expertise matters a great deal for how to make AI safe. It matters much less for deciding whether to worry, and the long technical answer to that question can leave the listener thinking the worry has been dealt with when it has not.

Related: The politics of AI risk, Superintelligence.