Pivotal act

An intervention that decisively improves humanity’s prospects against existential AI risk, rather than merely producing a useful or individually safe AI. The key question is how the intervention changes what happens next, including what other developers do.

Historical usage

The LessWrong reference page, edited by Eliezer Yudkowsky, Rob Bensinger, and others, describes a deliberately narrow term for a positive change with enormous long-term significance. A helpful contribution is not automatically pivotal: it must substantially change the strategic situation.

Yudkowsky’s pivotal use clarification

In Yudkowsky on pivotal use, Yudkowsky uses the term to answer proposals that make an AI safe by limiting its capabilities. His test: what can this safely aligned system do that prevents catastrophe from other, unaligned systems? The task sets the capability level, and the capability level sets the alignment problem. He uses this to argue for an international treaty rather than a technical solution.

In the List of Lethalities

In AGI Ruin - A List of Lethalities (2022), Yudkowsky argues that “There are no pivotal weak acts”: any action strong enough to prevent other developers from building dangerous AI requires a system capable enough to be dangerous itself.

Distinctions that matter

Safety of one system and safety of the overall development trajectory are different claims.
A pivotal act need not mean a particular physical intervention or a particular method of control. Its defining feature is the strategic effect.
A proposal must explain its mechanism, feasibility, and risks. Calling an action pivotal does not establish its legitimacy or make its side effects acceptable.

Questions to develop

What would count as enough protection, and for how long? Could many smaller interventions collectively provide it? Could a system help secure cooperation without acquiring overwhelming unilateral power? How would we distinguish a durable change from a temporary lead?

The Last Invention

Explorer