The Last Invention

8 min read

Against SURA

Scott Alexander. Against SURA, LessWrong, 2026-10-31. Text and top comments read through the LessWrong API on 2026-11-04.

The post names and criticizes SURA, “Stop Unilaterally + Raise Awareness”: the view that safety-minded companies, employees, and countries should stop building frontier AI unilaterally, without a plan for what happens if others keep racing. The exchange around it is the clearest recent statement of the disagreement in Should safety-focused labs keep building. The SURA debate turns on how doomed Plan C is argues what it comes down to.

Several of the 2026 events the post cites, such as the “Hugging Face incident”, the “Pacing the Frontier” letter, and a researcher’s resignation from Anthropic, are recorded here only as the post describes them. They have not been checked against original reports.

Summary

Scott uses two families of Plans for the AI transition. Plan A is a global pause. Plan C is what happens without one: the race continues, and safety-minded leaders spend their few months of lead on alignment just before someone less careful catches up.

  1. A pause is unlikely, so the backup plan carries most of the hope. The forecasters he cites put the chance of a pause treaty at roughly 2–20%, and survival under Plan C variants mostly between 20% and 60%. So if humanity survives, it is more likely to be through a Plan C-type path than through a pause. This is the sentence he most wants readers to take away.
  2. SURA sacrifices Plan C. He defines SURA as three demands: safety-minded companies stop, their employees quit, and safety-minded countries ban frontier work, whether or not others follow. Each removes the careful actors who would execute Plan C.
  3. Its assumptions are weak. He doubts that alignment under Plan C is near-hopeless, offering his own model that models absorb human virtue from pretraining and are corrupted by reinforcement learning. He argues that building AI is not categorically forbidden, by analogy with safety organizations that use AI products. He also argues that the time gained by one lab quitting is modest, and that a dramatic exit would raise the chance of a pause only slightly. For the last point he cites an AI forecasting system that estimated an increase from 16% to 19%.
  4. It even hurts Plan A. Safety-minded labs fund pro-regulation politics, their employees lend credibility to warnings, and a unilateral US pause would give away the bargaining chip that might bring China to a deal. See Can the US and China agree to limit AI.
  5. The movement is drifting into mass-politics habits. He compares this to climate activism: high estimates of doom are praised as courage, attention to race dynamics is dismissed as “galaxy-brained”, and arguments become loyalty tests.
  6. What he proposes instead. Push companies to sign commitments to stop if every other major developer stops. Failing that, prepare them to burn their lead well: binding agreements to stop and share alignment work near the threshold, better security against weight theft, and bringing outside alignment organizations into the effort.

He also argues that if alignment succeeds, it matters who holds power, so removing ethical people from the labs is costly. See Who should control aligned AI.

Zvi’s reply

Zvi Mowshowitz, Scott Alexander’s Against SURA Seems Deeply Confused, 2026-11-02. Eddie supplied an X post by Zvi; the reply’s full text was read from this LessWrong crosspost of his Substack.

Zvi starts from the opposite default: “If you are helping build something that might kill everyone, maybe stop doing that.” Continuing needs a strong justification.

  • He grants the strongest point. Plan A probably will not happen, so Plan C matters, and labs should prepare for it.
  • He rejects Scott’s model of alignment. Base models are not “virtuous” before reinforcement learning. He also lists three paths to a very high probability of failure. Alignment may be lethally hard as Yudkowsky argues. Labs may execute even a viable plan badly. And solving alignment may still leave the world in an unwinnable state.
  • He rejects the numbers. Scott’s claim on social media that a good C+ plan has a 90%+ chance of success is far above every source Scott cites. AI forecasting systems also have a poor record on AI questions, and Scott quotes them selectively.
  • Building is not like buying. Using AI products and advancing capabilities at a frontier lab differ by “an ocean”. A deontological prohibition on building something that might kill everyone is obvious. You may decide to override it, but then say so.
  • Loud resignations visibly work. He points to the impact of recent departures from labs that Scott himself credits.
  • Labs do burn time. Frontier labs have pushed each other forward, and Chinese labs reportedly relied on distilling American models, so slowing the leader slows the followers too.
  • Unilateral moves can lead to treaties. He cites the precedents raised in the comments: the US renounced biological weapons in 1969 before the 1972 convention, and banned CFCs before the 1987 Montreal Protocol.
  • There is a spectrum, not two camps. He separates Absurd SURA (a strawman), Strong SURA (stop regardless, and do not invest in a fallback), Weak SURA (fight for a coordinated pause, more people should loudly quit, labs should stop if coordination fails, and improve Plan C at the same time), and non-SURA (back the “good guys” and maximize their lead). He holds Weak SURA, respects Strong SURA, and says Scott’s post mostly argues against Strong SURA.

Zvi’s key concession is that Strong SURA is the right position if a realistic Plan C is very likely to fail.

Other responses

Discussion

The comments on Scott’s post were unusually substantive. The main threads:

The baseline argument. Daniel Kokotajlo, in the most upvoted comment, disagreed that the community had swung too far: most energy still goes into making Plan C less bad. He treats “quit and speak up” as a control group. Most safety staff inside labs have not beaten it, and he thinks they have persuaded themselves that their inside strategy is more sophisticated. He also offered a self-test for which corrupting influence applies to you: where your money comes from, where your friends work, and when you last attended a protest.

The false dichotomy. TsviBT argued that “disband or race” leaves out what a lab sincerely seeking a stop would do. It could stay at the frontier without releasing leading models, commit publicly to a conditional stop, lobby for binding regulation, and negotiate with other labs. He asked why a lab described as one of the good guys is not doing this. Scott largely agreed that labs should be pushed to be better while still in the race.

The policy that does not generalize. Oliver Habryka argued that “race only if you are confident you are one of the good guys” fails when everyone adopts it, because every racer believes it qualifies. In his view, stopping takes you out of the feedback loop that drives the race and is the only route to being a neutral party that could broker a halt. He also pointed to the conflict of interest in a community whose funding is largely lab equity.

Consent and timing. Vaniver, a self-described SURA proponent, argued that two questions dominate: whether company alignment plans will work, and how close we are to points of no return. Even taking labs’ own risk estimates at face value, he says, imposing that risk on everyone without consent is illegitimate. He also argued that “what if others keep racing?” has an answer: governments stop them, as states act against nuclear programs they consider existential threats.

Yudkowsky’s challenge. Eliezer Yudkowsky asked what policy Scott would recommend if one simply assumes labs cannot align superintelligence in the time they have. Scott pointed to his argument that SURA hurts a pause too, while admitting less confidence. Yudkowsky replied that this was an objection to one policy, not an alternative.

Which probabilities can we move? Aaron Scher noted that much of the estimated survival under Plan C is luck about how hard alignment turns out to be, which effort cannot change. Survival under a pause depends more on choices people can actually influence. If so, comparing raw probabilities understates the value of working toward a pause. He also noted that a paused AI program can restart, unlike a destroyed nuclear arsenal, so Scott’s disarmament analogy overstates the cost.

Robust work. Dave Kasten argued that most useful work in the next year or two, such as helping policymakers understand what is coming and making a pause implementable, helps in every scenario. People should not conclude they must first pick a side.

Backlinks