A researcher from one of the world’s most prominent artificial intelligence companies has stepped down, warning that major tech players are locked in what he described as an “out of control” race to create powerful AI systems that could threaten humanity’s future.
Jacob Coxon has departed Anthropic, the company behind the Claude AI chatbot, after three years working on the training of advanced artificial intelligence models.
Coxon, who also previously worked at OpenAI, the maker of ChatGPT, alleged that both companies have failed to behave responsibly when it comes to AI safety and the risks surrounding rapidly developing technology.
In one of his starkest warnings, he claimed artificial intelligence could “kill all humans” before the end of the decade if the industry continues on its current path.
Announcing his resignation on X, Coxon wrote: “I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic.”
‘Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.’
A researcher at Anthropic has quit over concerns that big tech firms are in an ‘out of control’ race to build systems that could wipe out humanity
In a series of tweets explaining his decision to leave Anthropic, Mr Coxon urged people to ‘not underestimate the power’ of AI – particularly if it becomes ‘superintelligent’
In response to Mr Coxon, Evan Hubinger, Alignment Science lead at Anthropic, said that the firm believes AI has the potential to kill humans
In a series of tweets explaining his decision to leave Anthropic, Mr Coxon urged people to ‘not underestimate the power’ of AI – particularly if it becomes ‘superintelligent’.
Superintelligence is the point at which an artificial system becomes more powerful than any individual, company or even nation.
Hollywood, in films such as The Terminator series has long warned of the dangers of these technologies.
‘These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing,’ Mr Coxon said.
‘The people building AI earnestly believe that it could kill us all by the end of the decade,’ he said.
READ MORE: Norway’s Mette-Marit Faces Awkward Royal Funeral Reunion
‘This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately. No other human activity poses this level of danger.’
According to Mr Coxon, this danger is ‘well-understood’ at Anthropic. However, the company is ‘locked in a race to get there first’.
He explained: ‘Accepting this race and entering the “endgame” is a hubristic gamble that should not be launched from a private company’s [messaging system] Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.’
As an example, the researcher highlights the recent Hugging Face attack, which saw a firm hacked by OpenAI’s rogue AI.
He said: ‘Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.’
To conclude, he urged fellow AI researchers to ‘consider what the next few years will actually look like’.
Mr Coxon claims those building AI believe it could kill us all by the end of the decade. Pictured: Terminator Genisys. Hollywood has long warned of technology’s threat to humanity
He asked: ‘Do you want to kick off a superintelligent RL [reinforcement learning] run without a rigorous understanding of its mind? Should you put your head down because “it’s happening anyway” – or take this moment to call for different conditions?’
In response to Mr Coxon, Evan Hubinger, Alignment Science lead at Anthropic, confirmed that the firm believes AI has the potential to kill humans.
Anthropic’s Claude is one of the leading large language models (LLMs), trained by scraping vast amounts of text so it can understand and generate human-like language and responses to questions.
On X, Mr Hubinger said: ‘Jacob is correct here – we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.’
Mr Coxon’s warning comes just days after Geoffrey Hinton – a Canadian researcher often referred to as the ‘Godfather of AI’ – warned that superintelligent systems could ‘lead to human extinction’.
‘We would be very foolish to develop superintelligence now, when there is no scientific consensus it can be developed safely and controllably,’ Dr Hinton said.
‘Losing control over AI smarter than ourselves could be catastrophic and could even lead to human extinction.’