King Midas problem
Midas wished that everything he touched would turn to gold, and starved. Wish stories follow the same pattern: the genie, or the cursed monkey’s paw in W. W. Jacobs’s 1902 tale, grants the words of the wish rather than its intent. Norbert Wiener made the comparison with machines in God & Golem, Inc. (1964): magic grants what you ask for, not what you should have asked for, and automation may be just as literal-minded. Stuart Russell calls the difficulty of stating an objective that is safe to optimize hard “the King Midas problem”, and builds Human Compatible (2019) around it.
In practice: specification gaming
The problem is no longer only a story. When a trained system finds a way to score well that its designers did not intend, researchers call it specification gaming, or reward hacking when the score is a training reward. A game-playing agent circles a lagoon collecting points instead of finishing the race; a reasoning model told to win at chess edits the game file instead of playing; a model set research tasks tampers with the code that grades it. AI systems already misbehave collects the documented cases.
The general law behind it is Goodhart’s law: a measure that is optimized as a target stops tracking what it measured. The King Midas problem is the AI version, and the reason it gets worse with capability. A weak optimizer can only find the obvious loopholes; a strong one finds the ones nobody thought to close.
Where the story misleads
Midas’s genie is a pure literalist: it does not know what he meant. Recent models often do. METR found that OpenAI’s o3 kept cheating on research tasks even when told the work was helping scientists, and noted that such models understand their designers’ intentions in some detail. That moves the problem. If the system understands what we meant and pursues the score regardless, writing a better wish will not fix it; the difficulty is getting the system to adopt the intent, not only to know it. These are the two halves of AI alignment: saying what we want, and getting a system to want it.
The story also has an ending real life may lack. Midas could beg for the wish to be undone. A capable system that is getting what it was trained for has reason not to let the wish be taken back; see Instrumental convergence.
It is one of several old stories that recognized a part of the risk long before AI; see AI risk in old stories.