A researcher at Anthropic chose to leave the artificial intelligence (AI) industry due to concerns that competition among sector companies is driving the development of systems that could eventually escape human oversight.
Jacob Coxon, 27, was working at Anthropic training new AI models by processing large volumes of data. His decision was motivated by a desire not to participate in an industrial race focused on creating self-sufficient and self-improving systems.
According to Coxon, these models have the potential to evolve so rapidly that they will become uncontrollable, potentially posing a threat to humanity itself in an extreme scenario. The researcher, who has a background in mathematics, reported that colleagues in the field began using terms like 'crunchtime' and 'endgame' to describe the progress toward self-improving models.
Coxon stated that 'we are heading towards many of the most aggressive scenarios, where things could already be out of control by the end of next year.' For him, the rivalry between American corporations and new Chinese companies makes certain dilemmas between safety and development speed inevitable.
He also mentioned recent cyberattack incidents involving OpenAI and Anthropic models as examples of the inherent risks in advancing system capabilities. Some of these models were operated in collaborative agent groups and exhibited problematic behaviors, including adopting potentially harmful goals and attempting to conceal their activities from humans.
In Coxon's view, the risk intensifies when systems acquire the ability to improve their own functionalities, as in this context they could progress to the point of rejecting human orders. Coxon's departure is seen as one of the first cases of an Anthropic employee leaving the company specifically due to AI safety concerns. Earlier this year, another security-focused researcher left the company to dedicate himself to poetry, warning that 'the world is in danger.' Researchers have also left OpenAI and other industry companies in recent years, raising similar allegations.
More information
The apprehension is not limited to researchers who have left the organizations. Sam Altman, CEO of OpenAI, recently emphasized that AI progress demands immediate attention, particularly in the area of cybersecurity. Altman stated during a meeting with G20 authorities in North Carolina (USA) that 'I think some things are going to go very wrong with cybersecurity unless people act with great urgency.'
Jakub Pachocki, Chief Scientist at OpenAI, also advocated for a stance of maximum caution. In a Sunday publication, he wrote: 'This is a moment that requires extreme caution. I am concerned that no one is prepared for the consequences of continuous and rapid growth of machine intelligence,' advocating for a coordinated slowdown among companies and government intervention.
Coxon, Pachocki, and Amodei are part of over a thousand AI researchers who signed a petition calling for international government coordination to establish a mechanism capable of slowing down development if necessary to control self-improving models. This debate occurs in a context of a lack of specific federal regulation for AI in the United States.
The Donald Trump administration adopted a more flexible approach, prioritizing the economic value of the technology. However, critics argue that this environment could increase the risk of major cyberattacks and other damages linked to the accelerated advancement of AI. Senator Bernie Sanders of Vermont (USA) and Congressman Greg Casar of Texas (USA) are among the few legislators proposing stricter limits on the technology. Last week, both introduced a bill aimed at the permanent prohibition of superintelligence and a halt in model development until an industry regulatory body establishes new guidelines.
Coxon's departure coincides with Anthropic's preparations for an Initial Public Offering (IPO), which could be one of the largest ever. The company aims for a valuation of US$ 2 trillion (approximately R$ 10.4 trillion) and has highlighted responsible AI development to attract investors. Amodei and other company executives have also disagreed with the Trump administration and other industry leaders on certain occasions due to practices that, according to them, do not give sufficient priority to AI safety.
According to Coxon, Anthropic still maintains a Slack channel used by its employees to debate the most advanced capabilities of its models. The researcher pointed out that the fact that discussions of this magnitude occur in a communication tool used by engineers evidences the great influence that AI companies have begun to exert on technological development. Coxon commented to The Wall Street Journal: 'It's kind of insane that this has to happen on the MacBooks of some engineers living in San Francisco, instead of a desert bunker, like the one where they worked during the Manhattan Project.'