As the artificial intelligence revolution speeds up, we are bombarded by both utopian visions and doomsday prophecies. It is difficult to assess the magnitude of the threat, because we have been conditioned to fear the wrong scenario. Science fiction has repeatedly warned us about the Big Robot Rebellion. In many science fiction novels and movies — like The Terminator and The Matrix — AIs and robots decide to seize control of the world, rebel against their human masters and enslave or destroy humankind. Such a thing is extremely unlikely to happen any time soon. The technology is just not there. At present, AIs are idiot savants. They may have mastered some narrow fields such as playing chess, folding proteins, or composing texts, but they lack the general intelligence necessary for highly complex activities like building a robot army and seizing control of a country. Unfortunately, the unlikelihood of the Big Robot Rebellion doesn’t mean there is nothing to fear. For it isn’t the killer robots we should be worried about; rather, it is the digital bureaucrats. Kafka’s The Trial is a better guide to the AI dystopia than The Terminator.
Humans have been conditioned by millions of years of evolution to dread violent predators like the one depicted in The Terminator. We find it much more difficult to understand bureaucratic menaces, because bureaucracy is a very novel development in mammalian and even human evolution. Our minds are primed to fear death by a tiger, but not death by document.
Bureaucracy began to develop only about 5,000 years ago, after the invention of writing in ancient Mesopotamia. But bureaucracy rapidly changed human societies in radical and unexpected ways. Consider, for example, the impact that written documents and the bureaucrats who wield them have had on the meaning of ownership. Prior to the invention of written documents, ownership relied on communal consensus. If you “owned” a field, it meant that your neighbours agreed it was your field, through both their words and actions. They didn’t construct a residence on that field, and didn’t harvest its produce, unless you allowed them to.