Can we learn from AI failures before they are fatal?

Will dangerous misalignment show up first in systems weak enough to fail visibly and survivably?

Why it matters

Engineering usually succeeds by iteration: bridges fall, planes crash, and the next design improves. Iteration only works if failures are survivable and informative. If they are, alignment looks like a hard engineering field. If the first serious failure is also the last, it looks like a problem that must be solved in advance.

The question matters politically too. Visible incidents, often called warning shots, are one of the main ways public opinion and governments change their minds. See Can AI progress be paused.

Warning shots cut both ways. A lab that prevents visible failures makes its own systems safer but may remove the incidents that would build support for a pause. Zvi Mowshowitz calls the argument for welcoming failures “galaxy brained” and rejects it unless the fallback plan is nearly hopeless. See Against SURA.

The case for one critical try

This is the first problem in AGI Ruin - A List of Lethalities.

• A system that cannot win has reasons to cooperate; a system that can win has different options. Behavior below the threshold may not predict behavior above it.
• A system that understands it is being tested can hide the failures we are looking for.
• Fixes found on weaker models may not survive large jumps in capability, especially if capability grows quickly. See AI takeoff.
• Warning shots may be explained away, patched, or kept from the public.

The case for many tries

• Capability may arrive gradually, giving many generations to study.
• Weaker systems already show recognizable failure modes, such as gaming their reward, which can be studied before stakes rise.
• Controlled tests, such as simulated environments where a model believes it could act undetected, can probe dangerous behavior safely.
• Each incident that does happen strengthens the case for caution, including among developers.

What would change minds

Whether models behave differently when they believe they are unobserved. Whether alignment fixes transfer across model generations. How quickly capability jumps between generations.

Related: How hard is alignment, Can AI automate alignment research.