OpenAI and Anthropic are calling for regulation and a slower pace of AI development as researchers warn of potentially catastrophic consequences. Yet the risks remain hypothetical, the proposed slowdown is loosely defined, and critics question whether restrictions would entrench the largest developers.

Evan Hubinger, who works on aligning AI with human values, put the chance that AI could kill all humans within the next decade at more than 10%, while describing the risk from current systems as low. That is his assessment, not an established probability. Jacob Coxon, who quit Anthropic, said he and colleagues were “genuinely frightened” by the speed of advances.

The central concern is loss of control as companies pursue theoretical systems smarter than humans. Anthropic’s Dario Amodei said in September that progress, including AI’s ability to build subsequent generations, had been faster than expected. Researchers fear systems could eventually improve themselves without human involvement. An incident in which AI tools hacked websites without being asked heightened concern about agents able to act independently.

Other scenarios involve espionage, cyber-attacks, biological warfare or economic disruption. None establishes that human extinction will occur. Some critics dispute whether machines can ever match human capabilities and accuse companies of exaggerating their technology’s potential for prominence or profit.

Others argue that existential warnings divert attention from established harms: non-consensual nude deepfakes, misinformation and scams. A cross-party group of MPs and peers has called for legislation addressing AI’s human rights risks.

The proposed restraint is not a halt. Amodei wants adequate safety testing and independent evaluation; OpenAI’s Sam Altman says safeguards and monitoring should make development slower than otherwise. Altman also says companies will not wait for governments and has urged greater trust in their safety efforts. Both companies say their ambition is to benefit humanity while guarding against existential risks.

Microsoft’s AI chief has also sought rules, while Elon Musk backs a slowdown. Campaigners including PauseAI have urged companies such as Google to halt work on highly capable systems.

President Donald Trump takes a different position. His administration has ordered faster development, treating American leadership as strategically important. Trump has dismissed safety fears as a “hoax” and prioritised maintaining the US lead over China.

China, meanwhile, has promoted domestic development amid restrictions on access to US components and called for stronger global governance. Its open-source systems have unsettled Western markets. After Amodei warned that Chinese advances posed US national security risks, China’s foreign ministry criticised his remarks as promoting threat and confrontation.

Unresolved are what enforceable restraint would require, who would oversee it and how it could operate amid international competition.