Neural Networks Are Already Clean

Nick Cammarata (Goodfire). Neural Networks Are Already Clean, 2026-10-15, revised 2026-10-30. An illustrated essay, about 65 minutes long, presented as “a working hypothesis”, with a technical companion paper to follow.

Cammarata helped found modern interpretability, working with Chris Olah on the circuits research at OpenAI. Here he argues that the field’s founding pessimism rested on misread evidence, some of it his own. Networks are not tangled knots that must be unpicked before they can be studied; their own neurons are mostly readable already.

The argument

The famous examples were misread. The field’s best-known “polysemantic” neuron, in the image model InceptionV1, seemed to respond to both cat faces and the fronts of cars. Cammarata now reads it as a detector of a V shape, a pointed bilateral form found in cat and fox faces, cars seen at certain angles, pool tables and bullet trains. Its strongest negative input is round, an O. The circuit he and Olah once presented as the first example of superposition, a car seemingly hidden among dog neurons, turns out to be cars and dogs grouped by shared pose. He knows of no second published circuit of that shape.

Messy layers came from bad tools and bad architecture. The visualization tool used to render what late neurons respond to was itself gaming its objective, drifting toward dogs because the training data is full of them. And InceptionV1’s last layer, once thought to have almost no readable neurons, becomes nearly fully readable in a modern network of the same size and data, because changes made to help networks learn, such as residual connections, also make them cleaner. Squeezing a layer down from 480 neurons to 10 did not produce mixtures of unrelated things; it produced coarser but still coherent neurons, such as one “curves and loops” neuron instead of ten curve detectors.

What remains is “recursive generalization”. A neuron that grows to cover related territory can end up covering things that no longer share its original pattern: a neuron for “iy” comes to fire on “diy” and then on “do it yourself”. This looks polysemantic but is ordinary generalization, followed one step at a time, and it can be watched happening during training. Superposition, he argues, should be the last explanation to reach for, not the default, because “something doesn’t make sense” is also what bad tools and unfinished explanations look like.

Language models are far more readable than believed. Anthropic’s 2023 study of a small model and his own automated project at OpenAI found almost no readable neurons. With modern tools and models, his team’s preliminary audit of Qwen3-32B finds about 90% of neurons have one main job, about 7% two related jobs, and only 2–4% are not understood, with similar results in models of up to trillions of parameters. Puzzling cases kept resolving: a “pr” neuron that also fired on random-looking web addresses was reading “pr” words encoded inside them.

Understanding compounds. Like a crossword, each neuron explained constrains its neighbors, so the work speeds up as the map fills in, and larger models, with narrower neurons, may be easier rather than harder. He suggests a “bitter lesson” for interpretability: scaling ordinary investigation by AI agents may beat engineering clever ways to decompose the network.

What it would mean for alignment

The most speculative step, which Cammarata labels as such, is that interpretability “might be far more tractable, possibly in the near term, than we’ve assumed”. He imagines a laboratory of automated scientists watching a network through training. Arguments that alignment is very hard often assume we cannot see inside models and must get training right in advance. If we can, he suggests, alignment might proceed by many simple, careful fixes, such as removing a training environment that turns out to be harmful, rather than by pushing gradients through the tools used to see.

Assessment

The essay’s evidence is strongest where it revisits specific published examples and weakest where it generalizes: the language-model figures are preliminary audits, by one team, of what counts as “one job”, and the companion paper with the details has not yet appeared. A neuron that is readable is also not the same as a model whose goals are readable; the essay is about the building blocks, and the hard alignment question is whether reading them reveals what a system is trying to do.

Responses

The essay drew more attention than most interpretability research, and a split between excitement about the headline and caution from specialists.

• Enthusiasm. Roon, of OpenAI, called it possibly “a major moment in scientific history”, news “so good even eliezer will have to lower his pdoom”, and said alignment could become “a ‘normal science’”. Sam Altman wrote that he was “very happy to see this”. Goodfire followed with a $10 million commitment to “a Human Genome Project for AI”: reverse-engineer a frontier-scale model and publish a living encyclopedia of how it works.
• Useful, but not the big picture. Neel Nanda, who leads interpretability at Google DeepMind, called it “interesting work” but not “a significant update to the big picture”. Studying neurons directly may well work for practical questions, such as why a model did something suspicious, and is far easier on frontier models. But he still thinks superposition is real: models hold far more memorized facts than they have neurons, and he expects the weak activations the essay sets aside to carry much of that information.
• The experiments do not replicate. Liv Gorton, in a detailed reply written with Cammarata’s input, argues that the essay refutes a non-standard definition of superposition, that its re-explanation of the famous cat-and-car neuron fails standard tests, and that her attempts to reproduce several experiments gave different results. Cammarata told her he plans further studies of that neuron. An informal replication on small language models found that the automated labeler called more random, untrained neurons “single-concept” than trained ones, a warning about how easily such labels flatter.
• Even if true, it is the start. Eliezer Yudkowsky said his objection was to the hype, not the researchers, and that he still supports the field. But he compared this stage of progress to setting out to “pry the panel off this machine and look inside”, having the screwdriver break, and announcing years later that the screwdriver is repaired. Finding single neurons for concepts is, he wrote, close to “the absolute first beginning of understanding how a mind is thinking”; real alignment work would mean predicting what a system will think “even as the thought departed from human”. And no level of interpretability, in his view, makes aligning superintelligence “normal science” rather than “maaaybe possible”.

Related: Seeing inside AI, Mechanistic shaping, How hard is alignment.