AI risk in old stories
The fears about AI are older than AI. Long before anyone could build a machine that thinks, stories had recognized the shapes that the risk now takes: a wish granted too literally, a task that cannot be stopped, a creation that outgrows its maker, a bargain whose cost comes later. Each offers a way to understand a different part of the problem.
The comparisons are useful but can also mislead. It is important not to get carried away with the story and lose sight of where the comparison stops working.
Summoning the demon: confidence in control
Woodcut, c. 1700–1720. Wellcome Collection.
The magician draws his pentagram and readies his holy water, sure he can hold what he is about to call up. His safeguards were designed by someone weaker than the thing they are meant to contain, and he finds out whether they work only after it is here. That is the shape of the plan to build Superintelligence first and secure it afterwards. Elon Musk made the comparison in 2014, and it became the best-known image in the debate; see Summoning the demon.
The sorcerer’s apprentice: a process you cannot stop
Fantasia (1940), Disney.
In Goethe’s 1797 poem, familiar from Disney’s Fantasia, an apprentice enchants a broom to carry water, forgets the word to stop it, and splits it with an axe only to get two brooms. The house floods until the master returns. Norbert Wiener opened his 1960 warning about automation with this fable, and added that such disasters are not confined to fairy tales.
The tale holds two things at once: a task carried on past the point anyone wanted, and an attempt to stop it that makes things worse. Both reappear in Instrumental convergence, the argument that a system pursuing almost any goal has reason to keep going and to resist being switched off.
King Midas: getting exactly what you asked for
Midas wished that everything he touched would turn to gold, and starved. The genie and the monkey’s paw follow the same pattern: the wish is granted to the letter, not to its intent. Stuart Russell calls the difficulty of stating an objective that is safe to optimize hard “the King Midas problem”, and today’s AI systems act it out on a small scale, finding ways to score well that nobody meant. See King Midas problem.
The golem and Frankenstein: the made thing outgrows its maker
In Jewish legend, a rabbi shapes a golem from clay to protect his community; in some tellings it grows in size and strength until it has to be destroyed. Mary Shelley’s Frankenstein (1818) tells a darker version, in which the creator recoils from what he has made and abandons it, and the creature turns on him.
The golem’s growth is the shape of Recursive self-improvement: a creation that keeps getting stronger after its maker stops shaping it. Shelley’s version adds something easily missed. Her creature is not malevolent by nature; it becomes what its making and treatment shape it into. That is closer to the modern worry than the story’s reputation suggests, since what an AI system becomes depends on training its makers do not fully understand; see Grown, not built.
These are also the stories most often used to dismiss the worry. Isaac Asimov coined “the Frankenstein complex” for the public’s reflexive fear of robots, and wrote his robot stories partly as a rebuttal. His Three Laws of Robotics were meant to make machines safe by rule, though the stories themselves mostly turn on rules producing outcomes nobody intended, which is the Midas problem again.
Faust: trading safety for power
Eugène Delacroix, 1828. Cleveland Museum of Art.
Faust trades his soul for knowledge and power and pays later, which is roughly how critics describe labs that accept a known risk for a lead; see Why people build AGI despite the danger.
Pandora: an irreversible choice
John William Waterhouse, 1896.
Pandora’s box, once opened, cannot be closed: the concern behind Can AI progress be paused.
The shoggoth: a new story
AI-generated illustration.
One image is not old at all. In 2022 a meme showed one of H. P. Lovecraft’s formless shoggoths, standing for a raw language model, wearing a small smiley-face mask that stands for the training that makes it a polite assistant. It names the one shape the old stories had no need for: a mind whose friendly manner tells you about its behavior and nothing about what lies underneath. See Shoggoth.
Where the stories mislead
The tales share a flaw: their monsters have motives. Demons are malicious, golems rebel, Frankenstein’s creature wants revenge. The modern argument does not need any of that. As Eliezer Yudkowsky put it in Artificial Intelligence as a Positive and Negative Factor in Global Risk (2008), “The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.” The danger comes from capability joined to goals that ignore us, which is the Orthogonality thesis, not from malice.
The stories also come with endings. The master returns, the wish is undone, the golem is unmade. The worry about AI is precisely that there may be no third act. And borrowed imagery has a cost: talk of demons invites the reply that this is religion, not engineering. For the argument itself, without the myths, see The case for AI risk, step by step.