Mechanistic interpretability

The attempt to understand what a trained neural network is doing by reading its internals: the neurons, the circuits they form, and the information flowing between them. Modern AI systems are grown rather than built, so nobody wrote down how they work; interpretability tries to work it out afterward, the way a biologist studies an organism.

Its promise for AI alignment is that it could check what a system is actually pursuing, instead of trusting its behavior. A model that behaves well because it knows it is being tested would look the same as an aligned one from the outside, but perhaps not from the inside.

Its central obstacle has been polysemanticity: many neurons seem to respond to several unrelated things at once, such as cat faces and the fronts of cars. The leading explanation, superposition, held that networks pack more concepts than they have neurons by overlapping unrelated ones. If so, the network’s own parts are a tangle, and researchers must first train a second model, such as a sparse autoencoder, to untangle it into readable features. Whether that explanation is right is now disputed; see Neural Networks Are Already Clean.

A second question is what happens when interpretability is used in training rather than only for inspection. Penalizing a model for an unwanted internal concept could fix the behavior, or could teach the model to hide the concept from the tools. See Mechanistic shaping.

Seeing inside AI is a reading path through the field, from its founding papers to the latest disputes. How hard is alignment treats whether interpretability can read goals at scale as one of the things that would change minds.