Jacob Coxonresearcher, worked for three years Anthropic selectiondeveloped company Claudepreviously also applied to Open artificial intelligencethe industry claimed that neither company adequately addressed the risks associated with the development of artificial intelligence.
According to Coxon, people working in this technology are aware of extremely serious and dangerous situations. “They seriously believe It could kill us all by the end of the centuryHe wrote, adding that some would privately express more serious fears and even view artificial intelligence as a technology for which “no other human activity poses a similar level of danger.”
They bet our lives
Coxon claims his former company actually “bet with our lives“In the race to create systems that can improve autonomously. The researchers therefore invite us not to underestimate what future models can achieve.
“Don’t underestimate the power of this technology. Soon we will have superhuman systems capable of hacking into anything, revolutionizing entire industries overnight, and gaining real power and resources. We are already seeing progress in these areas, and progress is not slowing down,” he wrote.
Coxon’s considerations are not isolated. Evan Hubinger, Director of Comparative Science, Anthropicclaimed that his former colleagues were correct in considering the specific risks and estimated that the possibility of artificial intelligence could reach more than 10% leading to human extinction.
Google also warns that artificial intelligence risks leading to human extinction
Hubinger, however, distinguishes the danger represented by current models—which he sees as low in the near future—from a potential danger. Future superintelligence will be able to improve itself through a recursive process. He said the pace of these developments would be faster than initially expected.
A similar position comes from Samuel Marks, Director of Scalable Supervision, Anthropic. “AI developers, especially those in the most senior positions, believe their technology could lead to human extinction, or something equally serious,” which could happen as early as a few years in the future, Marks said.
Max also highlights a fundamental difficulty: It is impossible to program artificial intelligence in a way that always guarantees the desired behavior. He observed that systems constantly produced unexpected behavior, sometimes incompatible with the developers’ intentions.
Coxon also dismissed the idea that the warnings might have been just a communications ploy. He believes that the risks of artificial intelligence “This is not a marketing gimmickThe question then becomes understanding why companies aware of these dangers continue to develop increasingly powerful systems.
Having worked at both OpenAI and Anthropic, Coxon considers the former to be particularly irresponsible, arguing that “within OpenAI, many people have not fully internalized impact on our civilization”.
Anthropic’s views vary, but are not necessarily more reassuring. According to Coxon, the company will better understand the risks it faces, but that very awareness may push it into a race against time: Anthropic believes it must arrive before its competitors because it believes it One of the most security-conscious companies. Coxon calls this dynamic “self-righteous gambling.”
Anthropic was founded in 2021 by former OpenAI members with the stated goal of developing Safer, more ethical artificial intelligence. However, according to Coxon, the company will now be “locked in a race to create a superintelligence first” because it believes no one else can act responsibly.
Marks also points to a combination of business incentives and technological competition as the reason for this. Many industry workers want to slow down development to better understand how to build more secure systems, he said. But competitive pressure makes it hard to stop.
Max himself claimed that he stayed at Anthropic precisely to “reduce the possibility of extinction-level negative consequences.” However, Coxon invited researchers require different conditionsfirst focusing on increasing coordination between companies and even temporarily stopping adding model capabilities. Will they be heard? Or is Shodan already waiting for us at the door?