Grown, not built

Ordinary software does what it does because a person wrote instructions for it. Modern AI does not work that way. Engineers write a learning procedure, choose the data and the goal it is scored against, and run it on enormous amounts of computing power. Training then adjusts billions of numbers until the system performs well. The abilities that result, and the way they are carried out inside, were never written down by anyone.

The comparison is with gardening or breeding rather than engineering: people set the conditions for growth and select for results, but they do not lay out the structure that emerges. Chris Olah’s version of the point, quoted by Dario Amodei in The Urgency of Interpretability, is that these systems are “grown more than they are built”.

Why it matters

• Nobody knows exactly what is inside. The builders can measure what a system does on the tests they think to run, but cannot read off how it does it or what it will do in untested situations. Mechanistic interpretability is the attempt to work this out afterward; see Seeing inside AI.
• Training shapes behavior, not intentions. Training rewards outputs that score well. Whether the system has picked up the intended aim, or a different one that happened to score the same, is not directly visible. This is the core of the AI alignment problem, and one reason Goodhart’s law bites.
• Surprises are normal. New abilities and new failures can appear without anyone having added them, which is why labs test models extensively after training and still find things later. See AI systems already misbehave.
• Fixes are indirect. A bug in ordinary software can be found and corrected in the code. A grown system is usually corrected by more training, which changes the behavior without guaranteeing that the cause is gone.

Limits of the metaphor

“Grown” does not mean uncontrolled. Developers choose the architecture, data and training goals, and those choices reliably shape what comes out; current systems mostly do what their makers aimed for. The point is narrower: the control is over the conditions and the results, not over the internal design, so confidence in a system rests on testing it rather than on knowing how it was built.

Related: How hard is alignment, Artificial general intelligence.