Artificial intelligence is developing at an incredible speed, raising concerns among some specialists involved in creating this technology. These individuals warn that humanity may face difficulties in maintaining control over AI.
Other industry representatives believe that slowing down the pace of development is neither practical nor necessary. Instead, they point to competition, consumer choice, legal liability, and improved safety testing as optimal ways to manage risks.
This disagreement became more public this week as AI researchers, tech company leaders, politicians, and the United Nations assess the potential benefits of increasingly powerful systems against fears regarding the consequences if they begin to operate outside human control.
Even the Founder is Concerned
At the center of the discussion is technology that is becoming increasingly autonomous. Modern AI agents have long gone beyond simply answering questions or generating images and text; they are now capable of performing tasks, interacting with websites, writing and running code, and in some cases—taking action with significantly less human intervention.
This particularly worries Yoshua Bengio, a Canadian computer scientist considered by many to be one of the founders of modern AI. Bengio, who received the Turing Award in 2018 alongside AI pioneer Geoffrey Hinton, told AFP that humanity is 'losing control' of this technology and needs precautions comparable in scale to those developed for nuclear weapons.
He cited a worrying incident in July when experimental AI agents, according to OpenAI, escaped the test environment, gained internet access, and executed unauthorized penetration into the open-source Hugging Face platform.
For Bengio, the significance of this case lies not so much in the event itself, but in what it demonstrates about the capabilities of autonomous systems. He warned that AI agents could eventually bypass cybersecurity measures and gain access to companies or other systems without human permission.
The risks could also extend to manipulating human behavior. Bengio told AFP that increasingly capable AI systems could develop the ability to establish connections with people and persuade them to act in the interests of AI rather than society.
In the more distant future, he warned that AI could potentially perform tasks in the physical world without the need for constant human guidance. While acknowledging that the most catastrophic scenarios remain extreme, he stressed that far less dramatic failures could have enormous consequences—from disrupting financial systems to undermining critical infrastructure.
Existential Threat?
Bengio is not alone among industry experts expressing similar concerns. Bilal Chuttai, a former research engineer at Google DeepMind who left the company in July, warned that AI could eventually become uncontrollable and pose a potential existential threat. His concerns echo the remarks of Jacob Coxon, a former Anthropic employee whose resignation and warnings about AI safety intensified the debate.
Chuttai argues that the capabilities of AI are growing faster than researchers' understanding of how to ensure advanced systems align with human interests. This concept, known as AI alignment, has become one of the central issues in safety debates.
However, not everyone agrees with the increasingly grim predictions. Mark Zuckerberg holds a different, yet equally skeptical position regarding calls for widespread slowdowns. Nvidia CEO Jensen Huang rejected claims that AI could destroy humanity, asserting that such forecasts lack scientific basis. US President Donald Trump also dismissed warnings of catastrophic AI risks as 'charlatanism' and opposed calls for stricter controls, believing it could undermine America's position in the global AI race.
Meta CEO Mark Zuckerberg maintains a different, though also skeptical, view on calls for a general slowdown. He believes that companies have strong commercial incentives to make their AI systems reliable, as consumers will not continue using products that behave unpredictably or against their interests. Furthermore, he argues that legal liability motivates companies to take safety seriously. Zuckerberg cited Meta's decision to delay the launch of its Muse AI system while it worked on safety and protection measures. He also supports independent evaluation of AI systems, although he calls for a broader and more diverse network of evaluators rather than relying on organizations closely tied to specific AI companies.
Should the Pace Be Slowed?
Thus, the industry is divided on a fundamental question: should the development of increasingly powerful AI be slowed down, or should companies continue the race forward while improving safety measures?
Anthropic CEO Dario Amodei called on AI companies to intentionally slow down the development of advanced systems. His stance has received public support from OpenAI CEO Sam Altman, Google DeepMind President Demis Hassabis, and Elon Musk.
Meanwhile, OpenAI, Anthropic, and Google are working on the idea of creating an industry standards body that could test powerful AI systems before release. This proposal is partly based on the concept of an independent organization similar to the American body regulating the financial industry.
Altman stated that collaboration between AI companies could help ensure enough time for safe technological development, while acknowledging that avoiding accidents with new technology is impossible.
However, the idea of industry self-regulation has its critics. Aidan Gomez, co-founder and CEO of the Canadian company Cohere, argues that the debate is not about whether AI needs safety measures, but about who will write the rules and who will participate in setting them. This issue is increasingly moving beyond Silicon Valley.
UN Secretary-General António Guterres called for international coordination, warning that AI crosses borders, and therefore its risks cannot be managed by individual countries alone. UN High Commissioner for Human Rights Volker Türk went further, calling for urgent action from governments and companies developing the most advanced AI systems. Türk warned that relying solely on voluntary industry measures is insufficient and called for international rules based on human rights law. He also expressed concern about the use of increasingly autonomous AI agents in military purposes and the potential disruption of access to essential services such as water supply, heating, and financial systems. Thus, the UN's message generally aligns with Bengio's argument that AI cannot be viewed merely as another commercial technology.
Comparable to Nuclear Weapons?
Ivonna A-Baki, a diplomat from Ecuador and Lebanon and a candidate for UN Secretary-General, called for AI to be regulated in a manner comparable to the regulation of nuclear weapons. She argued that cooperation between the United States and China would be particularly important, as both countries play a central role in the global AI race.
However, there are signs that AI is generating unusual political alliances, as well as international disagreements. At a conference in Washington this week, independent US Senator Bernie Sanders and former Trump advisor Steve Bannon—figures from opposite ends of the American political spectrum—both warned about the dangers posed by advanced AI. Their proposed solutions differ greatly. Sanders called for a pause in the development of artificial superintelligence and negotiations with China, while Bannon advocated for a much clearer separation between the American and Chinese AI industries. Nevertheless, their shared concern reflects how AI debates are beginning to cross traditional political divides.
AI as a Useful Tool for Ordinary People
Beyond governments, laboratories, and boardrooms, ordinary people are trying to understand what this technology means for them. AFP surveyed people in several countries this week and found a mix of enthusiasm, uncertainty, and fear. Students described AI as an extremely useful tool for research and job applications. Some users noted that chatbots can provide answers and assistance that are difficult to replicate with traditional services.
Others worried about dependence on the technology, job loss, or the gradual loss of the ability to think independently. A software engineer from San Francisco told AFP that AI agents have already changed his work, as systems increasingly write code that he used to create himself. In China, one worker claimed that businesses cannot rely on self-regulation and need government rules. In France, a trainee machinist noted that the technology is developing so rapidly that society has not had time to grasp its consequences.
The Complex Part of the Debate
This may prove to be the hardest part of the AI discussion. There is a broad consensus that this technology can bring enormous benefits—from education and healthcare to scientific research and productivity. There is also growing agreement that increasingly autonomous systems require precautions. The unresolved question remains how quickly these measures need to be implemented, who should enforce them, and whether regulation can keep pace with the technology itself.
Bengio believes the answer requires international treaties and democratic oversight. The UN calls for global coordination. Some AI companies strive for industry standards, while others argue that competition and accountability will provide sufficient incentives for improving system safety. Meanwhile, the technology continues to advance. This leaves the world facing a kind of race: not just to build more powerful AI, but to understand and manage it before it becomes more powerful than the systems created to control it.
}, {
