The case for AI risk, step by step
The worry that advanced AI could end in catastrophe is not one claim but a chain of them. Each step below builds on the one before. If you are not convinced, try to name the step where you get off.
Two things are worth knowing before you start. First, every step is a matter of degree. Doubting a step usually lowers the risk rather than removing it: a slow takeoff, say, gives more time to fix mistakes but does not by itself guarantee they get fixed. Second, the steps are not equally disputed. Most serious disagreement is about steps 2, 3, 5 and 7.
Who takes the conclusion seriously is a separate question, covered in Many serious people say AGI is dangerous.
1. AGI is possible
Claim. Nothing known about physics or biology stops a machine from doing the intellectual work a human can do. The brain is a physical system, and it is the proof that general intelligence can be built out of matter. See Artificial general intelligence.
Main objection. Intelligence may depend on something current methods lack, such as a body, lived experience, or a kind of understanding that pattern-matching on data does not produce. Possible in principle need not mean buildable with anything like today’s approach.
2. It may come soon
Claim. AI capabilities have risen quickly and steadily, the largest companies are spending fortunes with AGI as their stated goal, and many people building it expect it within years or a couple of decades rather than centuries. A risk that may arrive in our lifetimes is worth preparing for now.
Main objection. Benchmarks overstate real ability, progress may stall as data, compute and money run short, and confident predictions of human-level AI have been wrong since the 1960s. Builders also have reasons to talk up how close they are.
This site does not yet have a page on timelines.
3. AGI could become superintelligence quickly
Claim. An AI that can do AI research can help build its successor, and it can run as many fast copies as there are computers to run them. That feedback loop could carry capability far past human level in months or a few years rather than decades. See Recursive self-improvement, Intelligence explosion and AI takeoff; the classic review is Intelligence Explosion Evidence and Import.
Main objection. A feedback loop says nothing about its speed. Experiments, chips, power and testing all take time, and removing one bottleneck exposes the next. A feedback loop is not a speed estimate makes this case.
4. Superintelligence would be extremely capable
Claim. A system much smarter than us in science, strategy, persuasion and engineering, able to copy itself and act at machine speed, could outmaneuver any human organization, the way humans outmaneuver every other species. See Superintelligence.
Main objection. Raw intelligence may matter less than it seems. Humans dominate through accumulated culture, institutions and cooperation rather than individual brilliance, and a new AI would be born into a world already full of them. Matthew Barnett’s reply to the List of Lethalities makes this argument.
5. Its goals may not be ours
Claim. Being smart does not make a system want good things; very different goals can come with very high intelligence (the Orthogonality thesis, from The Superintelligent Will). And we do not write AI goals down; we train systems by rewarding outputs that look right, which shapes behavior without telling us what the system is really pursuing. Optimizing hard for a stand-in for what we want can produce something we did not want (Goodhart’s law, Strong Goodhart and overfitting). See AI alignment.
Main objection. Today’s models learn from vast amounts of human writing and absorb human concepts and norms, and ordinary training has made them mostly helpful and cooperative. The pessimistic arguments did not predict this. AI may also help solve the problem: Can AI automate alignment research. The whole dispute is collected in How hard is alignment.
6. A capable system with different goals would be dangerous
Claim. Almost any goal is easier to achieve with more resources and without being switched off or changed. So a capable system need not hate us to work against us; it only needs a goal for which humans, or our control over it, are in the way. Resistance to correction does not have to be programmed in. See Instrumental convergence.
Main objection. How strongly this applies depends on the system. An AI that answers questions or carries out narrow tasks, without long-term goals of its own, may have no reason to seek power. Systems can also be run under monitoring and restrictions that make it hard to act against their operators.
7. We may not get a second try
Claim. Engineering usually improves by failing: bridges fall and the next design is better. But a system capable enough to take control does not give us a chance to study the failure and try again. Lessons learned on weaker systems may not carry over, and a system that knows it is being tested can behave well for that reason alone. This is the first problem in the List of Lethalities.
Main objection. If capability rises gradually, each generation can be studied before the next is built, and visible failures in weaker systems serve as warnings. See Can we learn from AI failures before they are fatal, and The lethalities are a checklist, not a forecast for an assessment of the strongest pessimistic case.
8. And no one is reliably stopping it
Claim. If all the steps above are even plausibly right, the obvious response is to slow down until the problem is solved. But companies and countries are racing, each saying that if they stop, someone less careful will win. See Why people build AGI despite the danger.
Main objection. Rivals have limited dangerous technologies before, and frontier AI depends on a few chip suppliers and huge data centers that can be monitored. See Can AI progress be paused, Can the US and China agree to limit AI and Compute governance.
Where people get off
People who think the risk is small usually doubt step 2 (it is far away), step 3 (it will be gradual), or step 5 (training will give it good enough values). People who think the risk is very high usually think steps 5 and 7 together are the core: we cannot yet tell what a system wants, and we cannot afford to find out by failing.
If you get off at a step, the page linked there has the strongest case on each side. If you reach the end, the next question is what to do: see Plans for the AI transition and Should safety-focused labs keep building.