Researcher says AI has more than 10% chance of ‘killing all humans’

An artificial intelligence researcher quit his job at Anthropic on Tuesday and accused the company, and its chief rival, OpenAI, of acting irresponsibly, igniting a frenzy of concern on social media about the rapid pace of the technology’s development.

Jacob Coxon, who has worked as a researcher at both companies, said in a post on X that he resigned out of concern that Anthropic and OpenAI are “gambling with our lives.” He said the people building AI “earnestly believe that it could kill us all by the end of the decade.”

“Do not underestimate the power of this technology,” Coxon wrote. “These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.”

Coxon’s post, which has been viewed more than 70 million times, reflects a long-standing debate in Silicon Valley about whether AI can be safely developed and controlled. As Anthropic and OpenAI barrel toward potentially historic initial public offerings while releasing increasingly advanced models, many researchers are calling for a coordinated slowdown. 

OpenAI’s chief scientist, Jakub Pachocki, published a blog post on Sunday and warned that no AI company has “solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer.” In the AI industry, alignment refers to the work by AI developers to ensure that the system behaves in accordance with human values and intentions.

“I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established,” Pachocki wrote. “And I believe that international coordination on future AI development needs to become a top priority for governments around the world.”

Coxon’s post on Tuesday also struck a chord with industry researchers who are worried about recursive self-improvement, or an AI system becoming capable of designing and developing its successor without human intervention. Recursive self-improvement is not yet possible, but companies, including Anthropic and OpenAI, have warned that it would make it easier for humans to lose control over those systems. 

“Neither company is acting responsibly,” Coxon wrote. “They are racing straight to self-improving superintelligence.”

Evan Hubinger, an alignment lead at Anthropic, echoed Coxon’s comments in a post on X late Tuesday.

“Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is 10% within the next decade,” Hubinger wrote. “I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”

While extreme, concerns about the potential for AI to cause human extinction or other catastrophic events are not new in AI research circles. In 2023, for instance, prominent AI researchers and executives, including OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei, signed a statement that said “Mitigating the risk of extinction from AI should…