How Seriously Should We Take AI Doom Warnings?

Imagine an artificial intelligence making copies of itself. The first replicator assembles a copy in one thousand seconds, the two replicators then build two more in the next thousand seconds, the four build another four, and the eight build another eight. At the end of 10 hours, there are not 36 new replicators, but over 68 billion. In less than a day, they would weigh a ton; in less than two days, they would outweigh the Earth; in another four hours, they would exceed the mass of the Sun and all the planets combined.

This is the fate we face, according to many. Only notice that the paragraph above is in italics. That is because I did not write it.

It was written 40 years ago in a book published in 1986 and I have changed just one phrase, substituting “an artificial intelligence” for “a replicator floating in a bottle of chemicals”. The book was not about AI at all but nanotechnology.

Engines of Creation was written by Eric Drexler, and in it he coined the term “grey goo” for the fate of the world if self-replicating nanobots took off. Later he said he regretted the scare that his words sparked. Britain’s Royal Society, at the urging of the then Prince of Wales, wrote a lengthy report about the risk.

Yet nobody now talks about nanotechnology in such apocalyptic terms just as nobody says acid rain, ozone depletion, an imminent ice age or the Y2K computer bug are urgent existential threats. If you live as long as I have, you see many such scares come and go, and you get a tad cynical.

An X account called Pessimists Archive, run by Louis Anslow, collects such cases. He recently pointed out that in 2026 “the New York Times cites seriously academics who say AI might end the world”, and in 1881 “the New York Times cites seriously academics who say telegraphy might end the world”.

The paper opined that meddling with the Earth’s electric currents by building telegraph lines might freeze Africa and turn New York into a tropical jungle: “Such are appalling results which, according to our esteemed scientific contemporary, are to follow the girdling of the earth with a continuous line of telegraph.”

At this point my more responsible friends remind me that the boy who cried wolf was eventually right and there was a wolf. Also, AI is not like the other scares, they insist: this time is different. If I reply “they said that last time, too”, it cuts little ice, and of course they have a point. At some point there may be a wolf.

But this is when I get even more exasperated. Because my world-weariness at promises of theoretical future calamity is matched by my bafflement at complacency about a current, urgent calamity. Why, when a wolf did suddenly descend on the fold, did the entire bien pensant world dismiss it as an act of God?

I am referring to the Covid pandemic. In my view the evidence is now overwhelming that this began with a foolish experiment that went wrong in a laboratory in Wuhan — but even if you think that was only a small possibility, you should be really alarmed by the fact. If one virology experiment can kill 15 million people and do $45 trillion of damage to people’s livelihoods, then why have we largely ignored it while obsessing over some future potential threat from AI?

This point was brilliantly made recently by the celebrated venture capitalist Bill Gurley in a talk at the All-In summit in Los Angeles. “My p(second Covid) is way higher than my p(doom). And if their models are so ****ing smart, tell me what happened here.” To translate out of Silicon Valley-ese: we are facing a far greater risk of some virologist starting another pandemic than we face from AI. Yet nobody even wants to investigate, let alone prevent.

Gurley summarised in devastating detail how, when an accident happens, we usually leave no stone unturned to try find out whether it was somebody’s fault. When a building collapsed at Surfside in Florida, when two Boeing 737-Maxes crashed, when Hurricane Katrina overwhelmed New Orleans, when the Fukushima nuclear plant exploded, when the Challenger shuttle exploded, teams of investigators and journalists eventually found out that in all five cases these were man-made disasters, not natural accidents.

It took heroic determination to unearth the facts that, respectively, faulty design, faulty software, faulty levees, faulty flood protection and faulty cold resistance in the O-rings were the causes.

But in each of those cases the teams of relentless “searchers”, as Gurley calls them, had to fight like tigers to overcome powerful “blockers” from corporate and (yes) regulatory bureaucracies that did not want to find the truth. Yet those five events were pinpricks compared with COVID-19. Speaking of Richard Feynman’s battle to find out what happened to the Challenger, when Feynman was told he should try being nicer, Gurley said: “If you have to get through niceties and seek truth when seven people died, how are you supposed to behave when 15 million people died?”

The blockers did an amazing job with Covid. They scared off almost all journalists, politicians, spies and lawyers by saying the lab leak was a debunked conspiracy theory, and it was left to a few self-funded amateurs to do most of the work finding out what happened in Wuhan.

Western authorities not only failed even to try searching; they did nothing to pressure Chinese scientists: no hint that collaboration should cease till they came clean about what happened, for example. There has not been a ghost of an attempt to improve the monitoring of what happens in virology labs in future. The opposite, in fact: the WHO was warning about the risks of lab leaks before the pandemic but since — barely a squeak.

Yet we are all supposed to be engaged, dawn till dusk, in an obsessive debate about the vague — rarely specified — way in which AI might doom us at some rarely specified point in the future. Ah, say my responsible friends, but it is precisely through virology that AI might harm us. Imagine how much worse it will be when AI starts designing virus genomes! Maybe, but would it not be a good start to pay a smidgen of attention to what human intelligence can do and is doing?

“A lot has been said about AI bio trigger events that should lead to increased regulation or oversight,” writes my co-author, the virologist Alina Chan. “Yet, seven years since Covid emerged, the research pipeline that likely caused the pandemic is still legal to conduct outside of high containment without independent oversight.” To put it another way, we have just survived one shark attack and are swimming towards the shore in deep water, refusing to board life rafts, while discussing whether there might be a tiger where we land.

Matt Ridley is a British science writer, journalist and businessman. Author of The Rational Optimist and many other books of popular explanation, he writes on science, the environment and economics. This article first appeared on the websites of the Institute of Economic Affairs and the Rational Optimist Society.