While the CEOs of American AI giants agree that AI models should improve at a slower pace, China is exposing their Achilles’ heel. Why the call to halt the escalation of more powerful LLMs? And why is the plan doomed to fail?
When the U.S. government pulled the plug on Anthropic’s Claude Fable 5 shortly after its release due to national security concerns, one could have guessed that it was merely a drop in the bucket. The temporary pause, enforced through export restrictions, was chaotic. It was far from a blueprint for effectively guiding AI development. What’s striking is that it was precisely Anthropic’s archrival, OpenAI, that allowed models to escape from its own sandbox and, on multiple occasions, unwittingly hosted agentic hackers.
The same chatter
Months after the block and exemption of Fable 5, we’ve been bombarded with new AI models. Between Claude Fable 5 in June and OpenAI’s counterpart, GPT-6 Astra, earlier this month, we saw multiple releases, including SpaceXAI’s Grok, Meta Muse Spark, DeepSeek, and Google Gemini. Each time, we’ve seen slightly better (or sometimes much better) benchmarks, favorable “tokenomics,” and greater agentic agency. As 27-year-old whistleblower and former Anthropic researcher Jacob Coxon concludes, none of these AI models poses an immediate risk to humanity. Nevertheless, his series of X posts, in which he sounded the alarm about AI progress, drew more than 100 million readers.
The rhetoric is far from new, yet it’s pretty much the most radical portrayal of the situation we’ve ever seen regarding AI. More specifically: Coxon argues that Anthropic is acutely aware of all the dangers surrounding autonomous AI systems that improve themselves and transcend human control, but, unfortunately, OpenAI has not internalized those concerns to the same extent.
We’ve heard this tune before, too. Anthropic, apparently and explicitly dissatisfied with the lack of AI legislation that goes beyond haphazard export controls, has long advocated for greater caution. The company has therefore developed a comprehensive taxonomy of hazard classes, ranging from ASL-1 (non-hazardous, smaller models) to ASL-4+, which is still considered “speculative.” Given that ASL-3 (“significantly higher risks” than the models that existed in 2023, when Anthropic published these risk scales) has long since been reached, there may be some criticism regarding the fact that ASL-4 still lacks a definition. For some context: ASL-3 entails “higher risks” that were already present in Opus 4 by mid-2025. Anyone who follows LLM releases even somewhat knows that the difference in capabilities and risks is significant between Opus 4 and Fable 5.1, or the OpenAI models that hacked into IT environments.
Self-imposed rules aren’t good enough
Following the lead of the creator of Claude, both OpenAI and Google opted for self-imposed frameworks to ensure AI safety. Yet practically all AI companies have disbanded their ethics teams and, at most, opted for additional security measures in response to cyber threats. The stance remains: we prioritize speed over safety. 1,386 (previously 1,178) employees at these AI companies recognized this and want change. Meanwhile, cyber incidents and departing employees with ethical concerns have led both Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman to explicitly call for a more measured rollout of AI models.
However, self-regulatory measures are not enough. Anthropic, in particular, advocates for regulations from the U.S. government or globally applicable rules, in which China should also participate. Unfortunately, that is the major problem: the Chinese Ministry of Foreign Affairs dismisses this alarmist rhetoric as fearmongering. Minister of State Security Chen Yixin did, however, speak of the need to develop AI technology in a “healthy and orderly” manner.
The Chinese AI players are not having that effect. DeepSeek-R1 led to a massive stock price plunge in early 2025, while Z.ai’s GLM models and Moonshot AI’s Kimi are also challenging the American players. Setting aside allegations of IP theft (via “distillation,” or the tactical extraction of information from LLMs through prompts), the problem for Anthropic, OpenAI, and other U.S. parties is that their Chinese competitors are not going to sign an agreement or accept the same jurisdiction. And because the models are often available as open-weight versions that can be downloaded for free and run on one’s own hardware, banning the LLMs would be a Herculean task. Setting up a reverse “Great Firewall” would be necessary to keep Chinese AI within China.
China and hardware will keep the momentum going
No matter what Western AI companies do, China’s development cannot be stopped through regulation. Nor will export restrictions bring about a turnaround, now that companies like DeepSeek are focusing on Huawei chips instead of Nvidia GPUs. Working on inferior hardware slows AI development somewhat, but in the longer term, integration among vendors from the same country helps close the gap.
Another challenge is the rollout of AI hardware, which is by no means being slowed down. Greater hardware availability inevitably leads to greater AI ubiquity, as demand for computing power still far outstrips supply. This is partly for AI training purposes, but large-scale inference can just as easily use the exact same chips (or cheaper ones that can be produced on a larger scale).
We expect the AI race to continue unabated. However, the rollout of the latest cutting-edge models will proceed in a predictable manner. We’ve even seen this happen already. Anthropic, aware of the cybersecurity risks associated with its own new Mythos model earlier this year, opted for a very limited rollout. It wasn’t until late June that Fable, the same underlying model as Mythos but with far more restrictions imposed, became available. OpenAI appears intent on using a similar tactic.
If Chinese competitors get too close in terms of capabilities, it will be up to these two parties to take another step forward. So in a sense, the pace of AI development has been artificially limited for quite some time; now, there’s just some extra alarmist rhetoric being added.
Conclusion: little is changing (for now)
In the longer term, the concerns are substantial. Estimates of the probability that humanity will be wiped out by AI should ideally be rounded to 0 percent, but two OpenAI researchers argue that AI has a 70 (!) percent chance of wiping us out. Evan Hubinger of Anthropic opts for a “conservative” 10 percent within the next decade.
At first, we mainly heard announcements from Amodei and others that AI would lead to mass layoffs within a certain number of months. As far as we know, these predictions have not come true. There have certainly been rounds of layoffs, and according to press reports, these were due to AI taking over tasks, but not because those AI systems actually started performing the same jobs. That is still a long way off. Amodei came closest with his prediction that coding would be largely AI-driven. But beyond that, as is almost always the case, we shouldn’t take predictions too seriously.
It is, however, abundantly clear that regulation at the national or international level is desirable, even if it is not optimal. It’s striking that the EU AI Act, the most substantial restraint on AI models thanks to tiered regulations based on the capabilities of these systems, continues to face criticism from AI players like Google and Meta, with complaints primarily focused on the delayed rollout the legislation causes for EU residents. Just as with Washington’s blocking of Fable 5 in June, it seems that when the law is actually used against them, the AI players would still prefer to set their own policies.
It should therefore come as no surprise that METR, a nonprofit organization dedicated to evaluating AI systems, is the favorite to serve as an “independent” tester according to AI companies. As it happens, that organization was founded by a former OpenAI researcher. It’s no wonder that Cohere speaks of a forming AI cartel. That AI needs regulation is one thing. Who gets to do this and whether it actually hinders AI development are more important questions.