Intelligence and alignment

Separate three questions: what could an AI do, what would it pursue, and how quickly could that change?

Capability and goals

AGI and Superintelligence name capability thresholds. Orthogonality thesis questions the assumption that increasing capability brings human-compatible goals. Instrumental convergence asks why different goals could still lead to similar strategies.

AI alignment concerns whether the resulting behavior serves intended human aims. Goodhart’s law highlights one reason success on a training target can diverge from success in the world.

List of Lethalities is the strongest case that alignment will fail in practice; its page summarizes the argument and the main objections.

Coherent extrapolated volition is a proposal for what an aligned superintelligence should ultimately aim at, and why that is not a target for the first systems.

Pivotal act asks a further question: how would a safely aligned system change the risk from other developers’ systems? Yudkowsky on pivotal use explains the distinction between making one system safe and changing the overall trajectory.

Improvement and pace

RSI describes a feedback mechanism. Intelligence explosion and AI takeoff concern possible trajectories. Read A feedback loop is not a speed estimate for the distinction.

Compute governance is a seed for questions about the physical and institutional constraints on development.

Open questions

The disputed questions behind strategy, such as How hard is alignment and Can AI progress be paused, are collected in Cruxes.