A good future

If AI goes well, what are we steering toward? The question is harder than it sounds: a good future is hard even to describe, many futures that sound good are not, and many that are good would not last.

Hard to describe

Why Describing Utopia Goes Badly finds that written utopias read as dull, uniform, or alien. Yudkowsky’s Eutopia is Scary answers that a better future should feel alien, so a description that reads comfortably is probably wrong. Together they leave no safe option: too familiar undershoots, too strange repels.

The Worth the Candle epilogue is the depiction readers most often say gets through anyway, by showing a heaven that keeps some of its flaws.

Good-sounding, but not good

The Fun Theory sequence argues that a good life needs challenge and real stakes, not just comfort, and Value is Fragile that a future missing even one part of human values may be nearly worthless; compare Goodhart’s law. Deep Utopia asks what gives life meaning once machines can do everything useful.

Good, but not stable

Tegmark’s twelve AI aftermath scenarios are worth reading with one question: what keeps this world in place? Several pleasant ones depend on an AI that must stay restrained forever. Meditations on Moloch explains why a world of many competitors may drift into races nobody wants, and Who should control aligned AI covers the problem with the alternative.

What to aim for

Should the good future be designed or left open sets out both cases: pick a destination and hold it, or protect the ability to keep changing course. Coherent extrapolated volition is one attempt at a middle path. Telocracy is another: it designs institutions that hold to their stated purposes, and leaves the purposes to the people who join them. Test the answers against a concrete optimistic vision, Machines of Loving Grace.