Optimal Timing for Superintelligence

Nick Bostrom. Optimal Timing for Superintelligence: Mundane Considerations for Existing People, working paper, version 1.0, 2026. Read from the full text Eddie supplied.

When should superintelligence be built, if waiting makes it safer but people keep dying in the meantime? Bostrom answers with a series of simple economic models, deliberately limited in two ways. He counts only the interests of people alive today, not of the future generations who dominate most writing on existential risk, and he sets aside what he calls “arcane” considerations, such as simulations and digital minds. Within those limits, his answer is to get to AGI fast and then pause briefly.

The argument

1. Surgery, not Russian roulette. The status quo is not safe: about 170,000 people die every day, and other technologies bring their own risks. So the choice is between two risky paths, like a heart patient deciding whether and when to have a dangerous operation. “If anyone builds it, everyone dies” is matched by “if nobody builds it, everyone dies”, only more slowly.
2. Even high risk can be worth taking, for people alive today. If successful superintelligence brought mortality down to that of a healthy 20-year-old, life expectancy would rise from about 40 remaining years to about 1,400. On those numbers, launching now raises the expected lifespan of everyone alive unless the chance of extinction is above 97%. Bostrom presents this as conservative, since it ignores improvements in quality of life and more radical possibilities.
3. Waiting is worth it only in a narrow band. When waiting lowers the risk at some steady rate, long delays come out best only if the initial risk is very high and safety progress is moderate: fast enough that waiting helps, slow enough that the work is not done quickly anyway. Otherwise the best delay is short, usually under ten years. Discounting the future lengthens it; a better life after AGI shortens it; aversion to risk makes the decision more cautious without changing the picture much.
4. The best time to pause is just after AGI exists. Before AGI, safety researchers cannot study the system that matters. Once it exists, they can probe it directly and use it to speed their own work, so safety progress comes in a burst, then slows as the easy problems are solved. Time spent early in this period buys the most safety, so the models often favor a pause of months to a few years at that point, and sometimes favor going faster before it to reach the burst sooner.
5. Tests turn a date into a policy. If tests can show whether a system is dangerous, the right plan is not a fixed date but a rule: launch when tests keep passing, wait when they keep failing. That only helps if those deciding are willing to let deployment respond to what the tests show.
6. People’s interests differ. The old, the sick and the poor have more to gain and less time to wait, so a view that gives priority to the worse off favors earlier launch. Bostrom rejects the idea that years beyond about seventy add little value, noting that very old patients themselves often reject it.
7. The ideal is not a policy. No one controls the timing, and efforts to delay have effects of their own. He lists ways a pause could backfire: it comes too early and discredits later pauses; it produces safety theater; work moves to less careful actors or to the military; a pause enforced by a powerful apparatus pushes towards totalitarianism; hardware and algorithms keep improving, so the pause ends in an explosive “overhang”; or it hardens into a permanent ban. Of the forms a pause could take, he finds a leading developer burning its own lead the least risky, because it needs no enforcement apparatus and ends by itself when competitors catch up, while noting that today’s leads are a few months at most.

His summary of a practical upshot: “move quickly towards AGI capability, and then, as we gain more information about the remaining safety challenges and specifics of the situation, be prepared to possibly slow down and make adjustments as we navigate the critical stages of scaleup and deployment.”

Responses

The paper is written partly as a reply to Eliezer Yudkowsky and Nate Soares, whose book If Anyone Builds It, Everyone Dies calls for a global ban on advanced AI. Yudkowsky answered on X on 2026-02-13, replying to a post that quoted the paper’s life-expectancy threshold:

If you want to desperately grasp at immortality, sign up for cryonics instead of taking a 96% chance of killing the planet. How incredibly … selfish and evil would you need to be, to think that 96% was an acceptable gamble to take with your neighbors’ children’s lives?

His objection restates one he had made before the paper, to the general argument that everyone dies eventually without AI: “It is meaningfully different if everyone including the kids dies at the same time, meaning there’s no posterity” (X). Both points go to the paper’s chosen scope. Bostrom does not deny them; he sets the interests of future generations aside and promises a separate analysis from that perspective.

Olle Häggström, a mathematician who has written on AI risk for years and counts Bostrom among his strongest influences, replied in Full speed ahead! (2026-02-20), which also criticizes Dario Amodei’s essay The Adolescence of Technology. He makes three objections:

• The scope is a way out. Calling the analysis person-affecting “serves as a kind of deniability clause”: Bostrom can disown the reckless conclusions as mere calculation, while choosing to publish them now and to postpone the analysis that counts future generations. Häggström calls the focus on people alive today “very uncharacteristic” of the author of the founding texts of longtermism.
• Real preferences differ. The model counts mostly people’s own well-being. Asked whether living to 1,000 would be worth a high chance of dying in an AI catastrophe, most people would reject the trade outright, and overriding them in their supposed interest would be paternalism.
• No one asked. See Consent. Häggström holds that something this momentous needs the informed consent of at least a solid majority of the world’s population, and notes that the paper “says not a word about consent”.

A LessWrong comment sharpens the scope objection using the paper’s footnote 21, which allows the probability of doom to be read as the expected fraction of people who die. In the model, then, a 97% chance that everyone dies is equivalent to killing 97% of people now so that the other 3% can live for 1,400 years, a deal most people would find monstrous. And with even a slight concern for future generations, Bostrom’s version is worse than that deal, because the surviving 3% could at least have descendants. Supporting it therefore requires either purely selfish motives or accepting the monstrous deal.

Assessment

The paper is most useful as a framework: it makes precise why the value of a pause depends on when it comes, and gives the “pause late” position its clearest statement; see When should AI development pause. Its list of ways a pause can backfire is the most thorough on the site and supports the claim that a pause only buys time, with an addition: a pause that lets hardware and algorithms keep improving can make the eventual transition more dangerous, not just later.

Its conclusions rest on choices it states openly but that do much of the work. The first is the most important:

• Only people alive today count. On the impersonal view, where the lives of all future generations are at stake, extinction is far worse than the deaths of the present generation, and much longer delays can be worth it; Bostrom’s own “Astronomical Waste” (2003) made that case. The striking 97% figure answers a different question from the one most people worried about AI risk are asking: in the model, extinction costs only the lives of people now alive, and nothing for the generations who would never exist. A policy built on it would trade away the future for a better chance at longer lives today.
• The burst of safety progress is assumed. The case for pausing late depends on rapid safety gains once AGI exists. Whether those gains arrive, and whether an AGI capable of helping is also safe to study, is disputed; see Can AI automate alignment research.
• The decision-maker knows where it stands. The models assume someone can tell when AGI has arrived and can choose how long to wait. Bostrom acknowledges this in his closing section, but the hardest practical questions, how to detect the edge and whether anyone has a lead to burn, sit outside the model.

Related: Plans for the AI transition, Can AI progress be paused.