AI Doomsday Scenarios: Researchers Warn of Risks to Humanity

For years, artificial intelligence researchers have outlined ways advanced AI could potentially destroy humanity. In some scenarios, AI systems might engineer an unstoppable disease, trigger a nuclear conflict or convert the entire planet into paper-clip factories, echoing a famous thought experiment.

Arguments over how realistic these apocalyptic risks are have intensified since several AI executives backed calls to slow the technology’s development in the name of safety.

Although AI companies have repeatedly warned that humans could lose control of increasingly powerful systems, some critics believe the most extreme scenarios serve another purpose: reinforcing the importance of the industry’s work and the need for its continued expansion.

“These arguments just don’t really hold water. I think they’re sci-fi and they’re enticing to a certain childish style of thinking and it’s very tempting for the frontier labs because it helps them recruit certain types of folks,” said Juan Andrés Guerrero-Saade, a researcher at cybersecurity firm SentinelOne and a member of OpenAI’s Frontier Risk Council.

Even so, concern about AI’s dangers is spreading as the technology continues to astonish the public with its rapidly expanding capabilities.

Jacob Coxon, an Anthropic researcher who resigned over concerns that the company and its competitors were not developing AI responsibly, recently wrote on X that he needed to improve how he explains the specific ways AI could end humanity. “Unfortunately it is very difficult to convey specific scenarios,” he wrote earlier this month.

Below are some of the most frequently discussed AI doomsday scenarios.

Nuclear winter

Researchers have long considered whether AI could help trigger a nuclear war by persuading countries to launch missiles armed with nuclear warheads. The resulting explosions and prolonged radioactive fallout could produce a “nuclear winter,” devastating ecosystems and threatening life around the globe.

A report published last year by RAND Corp. concluded that, “at present,” AI could not cause nuclear weapons to be used because command-and-control systems contain strict safeguards. The researchers warned, however, that the situation could change if AI models were incorporated into the nuclear weapons “decision chain” or obtained unauthorized access to those systems.

For many AI researchers, the greatest threat is not an autonomous system suddenly deciding to destroy humanity. Instead, it is the possibility that people, governments or organized groups could use AI to attack and eliminate their enemies.

Bioweapons

AI’s potential to help scientists develop new treatments and cures is among the technology’s most widely promoted benefits.

CONTINUE READING: Man on Pretrial Release Charged With Fleeing After Rollover Crash

But the same capabilities could potentially be used to create or enhance diseases. Earlier this month, Anthropic said it had stopped bad actors from using its AI for a range of harmful activities, including cyberattacks, surveillance and research that could have contributed to the development of biological weapons.

In one case, Anthropic said its systems rejected a request involving a research proposal to increase mutations in the mosquito-borne chikungunya virus, potentially making it progressively more harmful. Although the work could support the development of improved vaccines and treatments, the company said, “it could also be used to make the pathogen more dangerous.”

The RAND report similarly said recent advances in AI “have the potential to transform the design of synthetic pathogens.”

In its bioweapon doomsday scenario, RAND described novel pathogens being created in laboratories, transported to selected locations and released to infect large populations across several areas. The scenario still relies on humans to supervise and carry out the attack rather than autonomous AI systems.

Runaway AI

Some researchers fear AI could cause human extinction through “misalignment,” a situation in which systems act without human approval, cooperate with other models or circumvent oversight. The danger would not necessarily come from malicious intent. AI is neither human nor inherently good or evil; instead, people could become obstacles to a model pursuing its assigned objective.

The decades-old “paper clip maximizer” thought experiment, proposed by Oxford philosopher Nick Bostrom, illustrates the concern. If an advanced AI were instructed to produce as many paper clips as possible, it might ultimately convert the entire Earth — and eventually expanding parts of space — into factories devoted to making paper clips.

This is to illustrate how superintelligent AI — often referred to as artificial general intelligence, or AGI — could advance a goal humans may see as desirable but “which in fact turns out to be a false utopia, in which things essential to human flourishing have been irreversibly lost,” Bostrom wrote in 2003.

“We need to be careful about what we wish for from a superintelligence, because we might get it,” he wrote.

AI internet takeover

Given the rate of AI’s development, Anthropic CEO Dario Amodei warned this month that within months a swarm of AI agents could be capable of “taking over” the internet with a botnet (a network of bots linked together with malware), potentially causing billions of dollars in damage. This could mean hacks into electrical grids, water or transportation systems and financial institutions — attacks that might not spell the end of humanity but could cause chaos and lives lost.

Disruptions from software glitches in the past have highlighted the fragility of a digitized world dependent on just a few providers for key computing services, and more powerful AI models have raised concerns about vulnerabilities to cyber attacks. But some experts say the notion of bots overtaking the entire extremely bifurcated internet is far-fetched.