AI catastrophe is overdetermined

Solving alignment would not be enough. Getting through the arrival of superhuman AI means clearing several hurdles, and failing any one of them could be enough for catastrophe. Optimism about one hurdle, even the hardest, says little about the outcome.

The case for AI risk, step by step follows a single route to disaster: an AI with the wrong goals. Laid out as a chain, it invites the reader to find a weak link and relax. This post looks at the problem from the other side. The question is not whether one particular story comes true, but whether every story is prevented.

Three ways out

There are three broad ways through, and each is a chain of requirements:

• Build it safely. Make powerful AI and get everything around it right: the five hurdles below.
• Don’t build it. Stop every frontier developer, and keep them stopped as algorithms and chips improve, until building becomes safe. A pause only buys time argues that this route can only delay the problem, not end it.
• Build up to the edge, then stop. Race to just short of dangerous AI, pause, and use the AI already built to solve alignment: “burning the lead”. This route needs parts of the other two: a lead long enough to burn, a way to tell where the edge is, and AI that helps with alignment without itself being the danger. See When should AI development pause.

Survival needs only one route to work, but each depends on many things going right.

The hurdles on the first route

Each of these has to go right, and each has its own page because each is a serious open problem:

1. Align the first very capable systems. Make them reliably pursue what their makers intend. See How hard is alignment.
2. Deal with everyone else’s systems. One lab’s safe AI does nothing about another lab’s unsafe one a year later. A safe system has to change the situation, for example by enabling a treaty that holds or defences that work, or the danger has only been postponed. See Pivotal act and Can AI progress be paused.
3. Prevent misuse. An AI that obeys its users perfectly is as dangerous as its worst user. It can help with weapons, attacks on critical systems or mass surveillance. See Misuse.
4. Keep power accountable. Whoever controls the most capable systems could gain power no one can check. See Who should control aligned AI.
5. Keep people in the loop. Even with no villain and no rogue AI, people could hand over one decision at a time until they no longer have a say. See Gradual disempowerment.

An example of how this works: suppose alignment is solved and the leading lab’s AI does exactly what the lab wants. If the lab cannot or will not stop a careless competitor, the world has gained a year. If it can, the means that let it do so are also the means for unchecked power, and hurdle 4 becomes the whole problem.

Why this matters

It changes how to read good news. A promising interpretability result, a well-behaved model or a credible safety plan is real progress on one hurdle. It is not evidence that the overall outcome is good. Many disagreements are between people who are talking about different hurdles.

It also changes what counts as a plan. A plan for the transition has to say which route it is on and something about every hurdle along that route, not only the one its authors work on.

The hurdles inside the hurdles

The fullest version of this argument is Nate Soares’s AGI ruin scenarios are likely (and disjunctive) (2022). He answers people who think doom requires humanity to “thread a needle”: “Saving the world requires threading the needle; destroying the world is the default.” His list stays inside the first two hurdles above and still finds many separate requirements:

• The world. A safe way to use the first powerful AI has to exist, and a leading developer has to know it and accept it. That developer needs years, not months, which means either a long lead or rivals cautious enough not to rush each other. Governments must not get in the way of whoever is doing it well.
• The research. Alignment researchers must spot every lethal problem early enough to solve it, much of it before there is a real AGI to study.
• The organization. The team that gets there first has to care about the right things. It has to tell a real safety solution from a fake one under heavy time pressure, and it must not split apart or leak the technology to others.

The first hurdle alone breaks down further still. The List of Lethalities gives dozens of separate reasons alignment could fail; The lethalities are a checklist, not a forecast weighs them.

Soares leaves misuse out deliberately: a world competent enough to clear all of this, he thinks, would probably handle misuse too. The list above adds misuse, accountable power and gradual disempowerment because they can go wrong even when alignment goes right.

Steven Byrnes makes the same move from another direction, in his reply to “AI is easy to control” (2023). Even if keeping AI under control turns out to be technically easy, someone will still build an AI that really wants something: for profit, for a cause, out of a belief that AI deserves freedom, or just to see what happens. Then the outcome depends on whether the controlled AIs can defend against the uncontrolled ones. An easy technical solution helps only those who use it.

The strongest objections

• The hurdles are linked. A world competent enough to clear one is more likely to clear the others, so multiplying separate odds overstates the danger. Soares agrees, and says this is where all his remaining hope lies: no number of extra hurdles can make survival less likely than the chance that he has misjudged humanity’s competence. That turns the list into a single question: will humanity handle AGI much better than it has handled other hard problems? His evidence that it will not is the response to COVID. He also notes a pattern in which technical people assume the policy side has the problem in hand, and policy people assume the reverse.
• Hard projects with many requirements succeed all the time. A rocket launch needs thousands of things to go right, and most launches succeed. Many requirements make failure likely only when no one gets to learn from mistakes, which brings the question back to Can we learn from AI failures before they are fatal.
• Success can have many routes too. Safety can come in layers: careful training, monitoring, legal limits, defences. Any one layer can catch a failure the others miss, which is the opposite of the many-ways-to-fail picture. The three ways out are alternatives in the same sense, and they help each other: a pause buys time for alignment, and aligned AI could help enforce a pause.
• Clearing one hurdle helps with the next. Aligned AI could help with the rest: designing defences, checking that treaties are kept, keeping institutions honest. See Can AI automate alignment research.

What would change this

Evidence on how linked the hurdles are. The case is strongest if the hurdles are independent: if, say, labs that do well on safety have no influence over governments and rivals. It weakens if progress on one reliably brings progress on others, for example if the tools for checking alignment also turn out to be the tools for checking compliance with a treaty.