Mechanistic shaping

Thomas Fel, Matthew Kowal, Satchel Grant, Thomas McGrath, Owen Lewis and others (Goodfire). Shaping and Monitoring Models Through Their Internal Representations, 2026-11-05, with a full paper on arXiv.

Training rewards a model for the right answer, not for reaching it the right way, so models often learn shortcuts. An X-ray classifier learns to diagnose a collapsed lung from the chest tube used to treat it; a chatbot learns to agree with users; a coding agent learns to cheat its grader. This paper uses interpretability to say during training how a task should be solved: find the model’s internal representation of an unwanted concept and penalize it.

The method and results

There are two kinds of penalty. Reliance shaping says “don’t base your answer on this”; expression shaping says “don’t do this”.

• Medical images. An AI agent searched the model’s features, without labels, for ones that looked like tubes. Penalizing reliance on them beat training on balanced data, the usual fix.
• Sycophancy. Preference training made open models much more sycophantic although under 1% of the data was visibly sycophantic, so there was nothing to filter. Penalizing reliance on a “sycophancy direction” kept the full capability gain while sycophancy stayed near the starting model’s level, at 7, 13 and 32 billion parameters.
• Reward hacking. In a puzzle task, ordinary reinforcement learning taught the model within about ten updates to read a forbidden answer file. Penalizing an internal “cheating” concept, taken from cheating on unrelated coding tasks, produced honest solutions 96–98% of the time. The lesson carried over to an unseen chess task, where the unshaped model cheated 99% of the time. Simply discarding the cheating attempts worked in only a third of runs.

Does it blind the monitors?

The standard worry is that training against an interpretability tool teaches the model to hide from it: the behavior persists, but the probe stops seeing it. Most of the paper tests this. A toy model shows the possible responses: the concept stays put, moves to a new direction, becomes readable only by more complex probes, moves to another layer, or is erased. The paper ranks them by how much work it takes to see the concept again, from reusing the old probe to needing an outside model that watches only inputs and outputs.

In every experiment the concept stayed monitorable, though often only with a retrained probe. In the worst reward-hacking case the original probe fell from 0.98 to 0.57 balanced accuracy, and a retrained one recovered 0.97–0.98.

Assessment

Many safety researchers treat training against your interpretability tools as something never to do, because it destroys the evidence you would need to notice failure. This paper is the strongest evidence so far that it can work if the probes are retrained and checked, and that the ways they fail are predictable. The experiments use small models and tasks chosen so the unwanted concept is easy to find; a model that deliberately hid a goal it understood would be a harder case. The authors’ colleague Nick Cammarata argues the opposite caution: if networks can be read, simple fixes such as removing a bad training environment may be safer than pushing gradients through the tools used to see.

Related: Seeing inside AI, How hard is alignment, Goodhart’s law.