Should the good future be designed or left open?

If AI goes well, should we steer toward a particular kind of world, or try to keep the future open so people can keep revising what they want?

Why it matters

The answer shapes what “success” means for the people building and governing AI. If a destination should be chosen, someone must choose it and something must hold it in place, which leads straight to Who should control aligned AI. If the future should stay open, the task is instead to protect the conditions for ongoing correction, and the question becomes whether those conditions can survive.

The case for leaving it open

• Nobody can describe the destination. Written utopias read as dull, uniform, or alien (Why Describing Utopia Goes Badly). A better future would probably shock us, so any description we find comfortable likely undershoots.
• Nobody needed to before. People in 1800 could not have described today’s medicine, and it arrived anyway, through a process that could notice and fix mistakes.
• Locking in a design locks in its errors. Every past generation held moral views later generations rejected. A future fixed to today’s best guess would freeze today’s mistakes too.
• Specific utopias leave things out. Value is Fragile argues human values have many parts; Deep Utopia shows how a world can satisfy most of them and still lack meaning. A designed future is a bet that the design got all of them.

The case for designing it

• Open does not stay open. Where many actors compete, each is pushed to sacrifice what all of them value to keep up; Scott Alexander’s Meditations on Moloch calls this force Moloch. Without some deliberate structure, an open future may drift into a race nobody wanted.
• Openness is itself a design. Keeping options open needs institutions, rules, and something that enforces them. Choosing those is already choosing a destination, only a less explicit one.
• Some outcomes are permanent. Extinction, or the lasting entrenchment of one group’s power, cannot be revised later. Avoiding them may require committing to more than “stay open”.
• A procedure can stand in for a picture. Coherent extrapolated volition proposes aiming at what people would want on reflection, which specifies a target without describing it in advance.

What would change minds

• Evidence on whether a world of many AI-empowered actors can stay stable without a dominant power, or tends toward races and consolidation.
• Whether human values seem to converge under reflection and discussion, which would make a procedure like coherent extrapolated volition more workable, or keep diverging.
• Concrete designs for institutions that let a future keep correcting itself, such as Telocracy, and whether any could hold against strong competitive pressure.

Read the A good future path for the sources behind both cases.