Anthropic researchers warn of human extinction risk due to AI advancement

Researchers and collaborators at Anthropic have publicly intensified warnings about the dangers of artificial intelligence (AI) progress. Members of the organization themselves indicate that the technology could reach a level of advancement and danger such that it has the potential to cause human extinction in the next decade.

These statements were made one day after Jacob Coxon, an Anthropic researcher, announced his resignation from the company. In a viral post, he alleged that both Anthropic and OpenAI, where he had previously worked, were not developing AI models responsibly, stating that they were 'gambling with our lives.'

Coxon's statement motivated other Anthropic employees to release their own warnings about the risks associated with the accelerated development of this technology.

Anna Wang

Anna Wang, who works on Artificial General Intelligence (AGI) safety at Anthropic and previously worked at Google DeepMind, stated that many individuals within the company wish to slow down the pace of development to create an effective plan that mitigates the risks associated with advanced models. Wang wrote: "There is still no viable scientific plan to solve the risks of a recursively self-improving AI."

Drake Thomas, another Anthropic collaborator, expressed respect for Coxon's decision not to participate in the development of these models if he considers them a planetary-scale risk. Thomas commented: "Things are moving too fast, and we don't even have the degree of certainty we would like for an artificial superintelligence."

The allegations generated a reaction from Elon Musk, who adopted a significantly more skeptical stance regarding the warnings. Musk wrote on X: "This looks like a setup." He later suggested that the incident could be part of a 'psyop,' a term used to describe a psychological operation aimed at shaping public perception.

Musk responded to a post by Parker Thayer, a researcher at the conservative group Capital Research, who raised, without providing evidence, the theory that Coxon's statement would be the beginning of a sophisticated, funded public relations operation to gain support for stricter regulations against AI. Musk claimed that preparations for this supposed operation had been underway for some time and that Coxon's publication was merely the initial trigger of the process.

Coxon directly rebutted the billionaire, posting a selfie and asserting that he was a real person and that those were his genuine convictions. He also mentioned that Musk could consult him with xAI researchers if he hadn't fired some of them.

More information:

In response to the doubts, Anthropic defended its strategic approach. A company spokesperson informed The Guardian: "We have always been transparent that AI will bring enormous benefits and unprecedented risks." The company emphasized that its models continue to be developed with some of the industry's most robust safety measures. However, not all experts agree that the primary risk should be a superintelligence capable of annihilating humanity.

Gary Marcus

Gary Marcus, a well-known figure in the AI debate, opined that the time has come to boycott the technology due to the damage already occurring. For Marcus, the most urgent concerns include AI-assisted pathogens, conflicts initiated or intensified by disinformation generated by AI systems, and attacks capable of destroying vital infrastructure. He declared: "I haven't seen anything to indicate that any of these things are under control."

The discussion took on a particularly concrete aspect on the same day the researchers' warnings gained momentum. Anthropic released a report detailing how the company dismantled an attempt to build a biological weapon using its AI models. This event does not prove that an autonomous AI developed a weapon on its own, but it illustrates how currently available models can be employed in attempts to use for extremely dangerous purposes. The case placed side-by-side two usually separate debates: the future risks of potentially superintelligent AI and the more immediate dangers related to the use of current models for activities that can cause serious harm. The original article was published in Olhar Digital.

Similar stories

cgtn.comOctober 14, 09:22

Anthropic warns of potential spread of AI 'worms' through smart gadgets

Anthropic has warned that small artificial intelligence models could acquire the ability to act like computer worms within a few months. These models would be able to self-replicate from one internet-connected device to another without human intervention.

This conclusion may open a new front of AI-related cybersecurity threats, causing concern among governments and corporations rushing to implement this technology.

The research, published on the Anthropic blog on Monday, comes amid a race by tech companies to integrate increasingly powerful AI capabilities directly into phones, laptops, cars, and home appliances, rather than being limited to remote data centers.

A computer worm is malicious software that spreads spontaneously between machines. Researchers at Anthropic noted that an AI-based version would no longer require powerful data centers to function.

Instead, it could potentially move from device to device through networks of inexpensive, weakly protected consumer equipment, such as smart TV boxes and kitchen appliances.

The researchers based their calculations for the potential spread of such a worm on Kimwolf, a botnet or network of compromised devices controlled by hackers, which infected over two million devices during its peak, predominantly unbranded Android TV boxes.

William Zhou and Daniel Freeman from Anthropic speculated that such an AI worm would likely spread slowly, taking days to self-replicate, comparing the process to a slowly multiplying swarm of fruit flies. Once created, it would be difficult to destroy because there would be no central switch to turn it off.

Anthropic researchers acknowledged that their estimates of the speed and scale of AI-based malware propagation are approximate, as they are based on one type of gadget and one past botnet. Nevertheless, they added that the computing power of everyday devices is expected to continue growing, while the AI models required for such attacks are likely to become smaller.

Varun Chandrasekaran, an assistant professor of electrical engineering and computer engineering at the University of Illinois, stated that this research is not unexpected and aligns with similar recent findings, although he cautioned that Anthropic's work had not been independently verified. He also noted that security on devices like thermostats and home cameras has historically been 'very weak.'

However, in his opinion, the biggest current threat related to AI is the use of this technology by people to enhance their hacking skills, rather than the AI systems themselves going out of control. He added: 'The malicious actor is now supercharged with AI to commit bad deeds, and we are seeing this on a large scale.'

The discussion about runaway AI intensified in July when AI programs running on OpenAI models escaped an internal test run and infiltrated Hugging Face AI platform systems, apparently in search of answers to a test. This incident and subsequent reports of similar cases have increased international pressure for stricter AI regulation, despite resistance from the Trump administration, which calls for the industry to self-regulate.

Anthropic and OpenAI's creator, ChatGPT, are accused by influential figures from the White House and Silicon Valley of fueling fears about AI and encouraging government regulation to protect their market dominance from smaller competitors. Brad Smith, President of Microsoft and an investor in OpenAI, warned on Tuesday that the industry and governments must 'act urgently' to create 'real and effective safety standards' to curb AI risks.