Summoning the demon

“Summoning the demon” is the best-known image in the AI risk debate. In October 2014 Elon Musk told an audience at MIT, “With artificial intelligence we are summoning the demon.” He went on to describe the scene from the stories, the man with the pentagram and the holy water who is sure he can control the demon, and added that it doesn’t work out. Several outlets mocked the line as hyperbole, and it has been quoted ever since, by supporters and critics alike.

The point is not that AI is evil. It is about confidence in control. The magician’s safeguards were designed by someone weaker than the thing they are meant to hold, and he finds out whether they work only after he has called it up. That is the shape of the worry about building Superintelligence first and securing it afterwards: the test of the containment is the event it was meant to prevent. The simple argument for AI risk is the strong one makes the same point without the imagery.

The image also has a cost. A demon is malicious by nature, and the modern argument needs no malice, only goals that are not ours; see Orthogonality thesis. And talk of demons invites the reply that AI risk is religion, not engineering.

It is one of several old stories that recognized a part of the risk long before AI; see AI risk in old stories.