The worry is older than AI
It is easy to assume that fear of AI is a recent invention, bred by science fiction or talked up by tech companies. In fact the central arguments were written down long before anyone could build a machine that thinks, some of them by the people who founded computing. What changed recently is not the argument but the machines.
1863: Butler and the evolution of machines
Four years after On the Origin of Species, the English writer Samuel Butler published a letter in a New Zealand newspaper, Darwin among the Machines. If living things evolve, he asked, why not machines, which were improving far faster than any animal? Day by day people were growing more dependent on them, and in time the machines might come to rule their makers. His half-satirical conclusion was “war to the death” against machinery, before it was too late. He later expanded the idea in his novel Erewhon (1872), whose people destroy their machines for exactly this reason. The fear itself is older still, in myths of servants that outgrow their makers; see Cautionary tales.
The name of the “Butlerian Jihad”, the war against thinking machines in the background of Frank Herbert’s Dune, is widely read as a nod to him.
1951: Turing expects the machines to take control
Alan Turing, who laid the foundations of computer science and proposed the test for machine intelligence that bears his name, also saw where success would lead. In a 1951 lecture, Intelligent Machinery, A Heretical Theory, he said:
“It seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. … At some stage therefore we should have to expect the machines to take control, in the way that is mentioned in Samuel Butler’s Erewhon.”
He offered it as a forecast, not a call to stop. But the claim at the heart of today’s debate, that machines smarter than us would end up in charge, comes from the founder of the field.
1960: Wiener and the wrong purpose
Norbert Wiener, the mathematician who founded cybernetics, the study of control and feedback in machines and animals, warned in Some Moral and Technical Consequences of Automation (Science, 1960) that a machine we cannot stop once it starts must be given exactly the right goal:
“If we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere once we have started it, … we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it.”
This is the alignment problem, sixty years before the name. “A colorful imitation” of what we want is what Goodhart’s law predicts a powerful optimizer will deliver.
1965: Good and the last invention
The statistician I. J. Good, who had worked with Turing breaking codes at Bletchley Park, made the argument that gives this site its name. In Speculations Concerning the First Ultraintelligent Machine (1965) he reasoned that a machine better than any human at every intellectual task would also be better at designing machines, so it could build a still better one:
”… there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.”
The first sentence is the origin of the Intelligence explosion and of Recursive self-improvement. The second is usually quoted without its last clause, which is the whole safety problem in miniature.
1993: Vinge and the end of the human era
The mathematician and novelist Vernor Vinge gave the idea its popular name in The Coming Technological Singularity, a 1993 paper for a NASA symposium. Its abstract begins:
“Within thirty years, we will have the technological means to create superhuman intelligence. Shortly after, the human era will be ended.”
Whether that ending would be good or bad for people, he left open, and asked how we might survive it.
Why it took so long to be heard
For decades these warnings could be set aside, because the machines they described did not exist and AI research kept failing to deliver even modest intelligence. The arguments did not need updating; they needed systems capable enough to make them concrete. When those arrived, the people building them began saying what Turing and Good had said; see Many serious people say AGI is dangerous.
Two lessons carry over. The worry does not come from fiction or from marketing: it is what careful thinkers concluded once they took machine intelligence seriously. And the hard part was named at the start. Good’s “provided that” and Wiener’s “purpose which we really desire” are still open problems; The case for AI risk, step by step sets out why.