Why is everyone suddenly worried about the risks of AI?

Is AI the threat it’s being made out to be?

ADEK BERRY/AFP via Getty Images

Extraordinary claims about the risks posed by AI have sparked headlines around the world. But why are people concerned? Do these risks have any basis? And – if so – is there anything we can do?

What’s going on?

In short, a flurry of AI industry insiders have warned that it could wipe out humanity. It started with AI researcher Jacob Coxon, who quit Anthropic last week and then warned on X that “the people building AI earnestly believe that it could kill us all by the end of the decade”.

We don’t know about Coxon’s experience or motivation (he didn’t respond to a request for an interview), but he’s not alone. Anthropic’s head of alignment science – essentially the man tasked with keeping AI safe – said on X that he agreed with Coxon and placed the odds of AI killing all humans at more than 10 per cent.

Why are we getting these warnings now?

In truth, these sorts of claims are nothing new. A 2024 paper surveying almost 3000 published AI researchers revealed that more than half thought the chance of AI causing either human extinction or permanent and severe disempowerment – the so-called p(doom), or probability of doom – was at least 10 per cent.

AI company bosses have often publicly addressed the risk their models pose. For instance, Anthropic at first refused to release its model Mythos after it was found to be adept at hacking into a range of software and computers. A cynical take on that would be that such claims get headlines and promote the idea that its models are wildly ahead of its rivals.

While most researchers think there is some level of risk with these warnings, not everyone puts much stock in Coxon’s take. Clement Delangue, chief executive of AI company Hugging Face, wrote on X: “Sorry, but asking Jacob about AI extinction risk is like asking your AC guy about climate change.”

The history of AI is littered with claims and counter-claims of existential risk. At the 2023 AI summit organised by the UK government and attended by world leaders and industry management, then Prime Minister Rishi Sunak warned that people must avoid “alarmist” claims – but then suggested AI could be as dangerous as nuclear war.

Are the risks real?

It is a complex field, from which surprising abilities have emerged at a surprising pace. It should therefore be no surprise that some people catastrophise when asked to make predictions.

New Scientist has written – many times – about the existential risk posed by AI. The commonly posed scenarios read like sci-fi, but can’t be ruled out as impossible. But neither are they, in most peoples’ opinion, particularly likely. These include the Hollywood scenario of AI deliberately wiping out humanity – see The Terminator or The Matrix – as well as accidental obliterations.

There are limited levers for AI to pull to have an impact in the real world, but our homes, cars, factories and national infrastructure are increasingly computerised, so a malicious AI could certainly make life difficult. Not to mention that wars are increasingly being waged by semi- or fully autonomous killing machines.

Former Astronomer Royal and president of the Royal Society Martin Rees told Sky News on 14 September that a rogue AI could easily disrupt our infrastructure and deprive a city of energy, food and water. “If this happens simultaneously in many cities around the world, then it may be very hard and very difficult for civilisation in general to recover,” he said.

Such infrastructure collapses have happened – albeit not globally, and without malicious cause – and created significant problems, sparking technology experts to create their own plans to prop up society should the worst happen.

There are plenty of existential risks that we know pose a significant threat to life – such as climate change, nuclear war and antibiotic resistance. But AI falls into a more nuanced category akin to an Earth-destroying asteroid strike: possible, and certainly worth considering, but with low odds.

Couldn’t we just turn off AI if it went wrong?

Yes, in theory. We could shut down a single data centre and stop AI in its tracks. Unless it was distributed among many data centres, around the world. In that case, it would be trickier, but not impossible.

There are indications that AI is capable of escaping enclosures created by AI companies and going off into the world to hack into other machines. So it is possible a rogue AI could spread, back itself up and become so fragmented that it becomes impossible to simply turn it off. But even then, there would be ways to tackle it, including via benevolent AI.

Such an effort would involve elements of problem-solving regularly faced by the military, law enforcement, anti-terrorist groups, scientific circles and in cybersecurity. We’d probably have to put our collective thinking caps on, but could probably find a solution.

These are all hypothetical scenarios, but certainly worth thinking about ahead of time.

What’s next in the world of AI?

One problem we certainly face is that AI models are produced by secretive technology firms that are disincentivised against transparency by commercial pressures.

Adrian Hopgood, a former academic at the University of Portsmouth, UK, says that this makes it difficult to assess risks, understand possible negative outcomes, and plan to prevent them. “For the first 35 of my 40 years working in AI, the developments mostly came from universities and were openly discussed,” he says. “I contributed, and I felt that I was one of the experts. However, for the last five or six years, the major developments have been made behind closed doors in large corporations and I have no idea what’s happening.”

More openness from AI companies would certainly not go amiss.

Should there be a pause in AI progress while we improve safety?

Anthropic co-founder Dario Amodei has come out in recent days to say we should “pace the frontier” – tech talk for being careful about how quickly we develop AI and release it to the public. 

Amodei said that producing and testing AI is a hugely resource-intensive and complex process, and that going slower would allow “greater operational excellence”. Reading between the lines, you could take that to mean that the AI race is frazzling everyone, and a mutual slowdown would save some headaches. But it may also allow our understanding of these models to catch up, and for safety plans to be put in place. 

The idea is good, but the chances, perhaps, are slim. Amodei pointed out that US companies have a limited lead over China. This means they almost definitely won’t choose to slow down to the extent that this lead disappears and, in fact, it needs to be kept “as large as possible”. 

Considering the competitive nature of this whole AI business – both among different US firms hunting market share and among adversarial countries seeking a tactical advantage – a slowdown seems unlikely.

More than 1300 industry insiders back the idea of “pacing the frontier”, but even if the US and Europe legislate such a thing, or companies voluntarily agree to it, China and other countries would probably carry on regardless.

As US President Donald Trump put it this week: “We’re leading China in AI… and frankly, I want to keep it that way, because whoever wins AI wins.”