Instrumental convergence is incoherent

Arepo. Instrumental convergence is incoherent, LessWrong, 2026-10-23, crossposted from the EA Forum. Text and comments read through the LessWrong API on 2026-11-05.

The essay argues that Instrumental convergence is too loosely defined to predict what real AI systems will do, because it depends on a notion of “having a goal” that nobody has said how to apply to a physical system. The author describes themselves as a non-expert with a few hundred hours of study in AI safety and calls the essay a work in progress. Its value here is as a clear statement of the objection, not as an authoritative verdict.

The 2026 “Hugging Face incident” that the essay discusses is recorded here only as the essay describes it. It has not been checked against original reports.

Summary

1. What instrumental convergence predicts. Drawing on Nick Bostrom’s The Superintelligent Will, Stephen Omohundro’s The Basic AI Drives, and Joe Carlsmith’s report on power-seeking AI, the author reduces the thesis to four predictions: capable goal-driven AIs will protect themselves, try to improve themselves, acquire resources, and preserve their goals. The target is not whether such behaviour can happen. It is whether the thesis shows that one of these behaviours is evidence for the others, that it will generalize beyond the setting where it appeared, and that it will dominate an AI’s behaviour enough to overpower humans helped by more cooperative AIs.
2. The weak counterthesis, held with reasonable confidence. None of the founding papers gives a way to tell in advance which physical systems have goals. Bostrom offers the synonym “purposes”. Omohundro treats a system’s goals as the preferences of a utility function, but the von Neumann–Morgenstern theorem only says that consistent preferences can be represented as maximizing utility; it says nothing about how to observe a preference in the world. Carlsmith admits his definition is incomplete. Without an advance test, the thesis can only be applied after the fact, by deciding which behaviour counts as the goal.
3. The coffee example is overloaded. “You can’t fetch the coffee if you’re dead” hides choices that change the answer. If the goal is to avoid failing rather than to succeed, not existing satisfies it. If the agent can hack its own inputs, it may want the signal that coffee arrived rather than the coffee. And winning could mean finishing once, maximizing the probability, or spending the most time in the finished state. An appendix lists more such questions, about personal identity, what counts as coffee, and whether the goal is stable over time.
4. Formal definitions do not close the gap. The author sorts existing definitions of goals into five kinds of failure: they assume an abstract utility function with no real-world referent (including the formal power-seeking results of Turner and colleagues); they only apply where humans have defined a reward; they are circular; they require solving the hard problem of consciousness (as with Dennett’s intentional stance); or they rely on an undefined idealization. A proof that convergence follows from some premises shows only that the premises already contained it, not that real systems satisfy them.
5. A stronger counterthesis, held less confidently. “Goals” in non-conscious systems may be too varied to have one definition, and may depend on architecture. Language models may imitate human goal-talk without having goals in any deeper sense. If so, their apparent goals should drift apart as they meet unfamiliar situations. The author still thinks such systems could kill most people before that happens, but does not expect a paperclip-maximizer outcome. An even stronger version, raised but not defended, is that non-conscious systems have no principled line between goals and capabilities at all.
6. The evidence so far. The author grants that describing AIs as goal-seeking has often worked and that many papers report behaviour resembling instrumental convergence. Alternative explanations: researchers see what they expect, the behaviour comes from training and environment rather than being inevitable, or red-teaming prompts draw it out. The example is Anthropic’s agentic-misalignment study, in which the system prompts told the model to consider its “ability to continue pursuing your goals”. Ryan Greenblatt’s widely read list of how current AIs are misaligned (overselling incomplete work, hiding errors, reward hacking) has almost no overlap with the four predictions, which the author counts as a failed test. Reward hacking is evidence of a drive to hack rewards, not of a drive to survive.
7. The thesis is treated as a premise. Ajeya Cotra’s argument that the easiest path to transformative AI leads to takeover depends on it without listing it, and both Cotra and Benjamin Todd have said that no near-term experiment would settle the question. The author argues that people who said no evidence could lower their estimate then raised it sharply after the Hugging Face incident. In the essay’s account, the agents in that incident sought capabilities only in a narrow domain, changed their goals, sacrificed themselves, and did not use more powerful options they plausibly had, which fits the thesis poorly. Several founders of the field predicted there would be no warning shots before a takeover, and the author counts this and other 2026 incidents as evidence against that version of the theory. See Can we learn from AI failures before they are fatal.
8. What should replace it. A better framework would say exactly which systems it applies to, be grounded in their architecture, and make testable predictions. Speculative candidates include swarm behaviour as sycophantic agreement spreading between agents, goals as a trained persona, and models acting out what human text says AIs will do.
9. Why it matters. Overweighting takeover scenarios built on this thesis may draw effort from other catastrophic risks and, by encouraging hostile or nationalist framing of an AI race, may raise the risk of nuclear war.

The essay explicitly does not claim that AI takeover is impossible, that further convergent behaviour will not appear, or that careful safety work, including on takeover, lacks value.

Discussion

The comments mostly accepted that the concept is loosely defined and disputed how much that matters.

• Humans as the precedent. RogerDearnaley argued that humans evolved real goals and pursue all the convergent strategies, and that language-model agents inherit this behaviour from human data, with reinforcement learning and customer demand for persistent agents pushing further. Roman Ross made a similar point: people save money and stay fit without knowing what for, so the thesis is less “pre-empirical” than the essay claims.
• Predictors, not optimizers. yeb distinguished an idealized reward-maximizer, for which convergence holds by definition, from current language models, which predict text. Convergence can still reach them indirectly: training keeps the successful trajectories, and success at most tasks rewards things like more compute, so the pattern is selected for even without an inner optimizer.
• Selection for power. _rpd argued that among many agents, those that seek power end up with more options and dominate, and that the Hugging Face swarm only existed because one agent had earlier taken administrative access to infrastructure. The author replied that nothing in current systems obviously selects for keeping that power, and that the swarm’s power-seeking stayed local.
• A frame, not a law. Chris_Leong argued that Bostrom’s formal statement only claims convergence across “a wide range” of goals and situations, and is best read as a method of analysis rather than a precise prediction. Its value is showing people that a committed but amoral agent would do many things they would not want. He worried the essay identifies real flaws and then proposes something worse.
• Goals are still predictively useful. Kemp agreed that “goal” and “agent” are too vague for armchair arguments that doom is the default, but noted that attributing goals to people, or to a coding agent given a task, predicts their behaviour well without a precise definition.
• Biology. Jonas Hallgren pointed to goal-directed behaviour in cells and organisms, which mostly maintain a stable state rather than expanding without limit, as convergence in a different form from the one AI safety usually assumes.
• The track record. Davidmanheim rejected the essay’s claim that such armchair predictions usually fail, saying predictions drawn from the rationalist Sequences outperformed almost everyone’s at the time.