Anthropic AI safety researcher quits, warns 'the world is in peril'

An AI safety researcher at Anthropic, Mrinank Sharma, has resigned with a resignation letter shared on X warning that "the world is in peril." He led a team at the company that researched AI safeguards, including work on why generative AI systems tend to flatter or agree with users rather than push back, efforts to combat AI-assisted bioterrorism risks, and research into how AI assistants could make people "less human."

Sharma said the danger was not limited to AI or bioweapons but came from "a whole series of interconnected crises unfolding in this very moment." He said he had "repeatedly seen how hard it is to truly let our values govern our actions," including at Anthropic, which he said "constantly face pressures to set aside what matters most." He plans to pursue a poetry degree and writing, and said he would move back to the UK and "let himself become invisible" for a period of time.

Sharma's departure follows a period in which Anthropic itself has issued unusually blunt warnings about its own field. In a blog post titled "When AI Builds Itself," the company said frontier AI systems may be approaching the point of recursive self-improvement, meaning they could design and build their own successor systems with little human input. Anthropic said this could raise the risk of humans losing control over AI technology, and proposed that the world's leading AI labs coordinate a slowdown or temporary pause to let society and alignment research catch up.

As evidence that the human role in AI development is shrinking, Anthropic pointed to its own coding practices: Claude now writes more than 80 percent of the code merged into Anthropic's systems, up from low single digits before the company released Claude Code in early 2025, and its engineers ship roughly eight times as much code per quarter as they did a few years earlier. The company said, "We are not there yet, and recursive self-improvement is not inevitable, but it could come sooner than most institutions are prepared for." It floated a "global coordination mechanism," loosely modeled on arms-control agreements, that would require participation from leading labs along with a credible way to verify that they had actually slowed down.

Anthropic's research institute plans to explore the issue with outside collaborators over the coming months and "take actions" toward building the systems needed for a credible slowdown, though the company did not specify what such a mechanism would look like in practice. The proposal also comes against a backdrop the company itself set out in earlier writing: Anthropic has said that scaling laws, the finding that AI systems get predictably smarter as they are trained with more computation, suggest very rapid AI progress is likely to continue rather than plateau, potentially bringing broad human-level or greater capabilities within the coming decade.

Not everyone is convinced by the pause proposal. Noah Giansiracusa, an associate professor of mathematics at Bentley University, said he does not think it is "a genuine call to slow down," arguing that Anthropic's chief executive, Dario Amodei, "wants to keep going full speed ahead." Giansiracusa also called a coordinated pause practically impossible, saying there is "zero chance there will be a slowdown," citing competitive pressure from other companies and countries. He argued that the evidence Anthropic cited, more code being written by AI, shows the technology is helpful rather than signaling what he called a major leap.

Mark Riedl, a professor at the Georgia Institute of Technology, wrote on Bluesky that "the big AI companies are all jumping on the 'recursive self-improvement' hype train," suggesting skepticism that the concept is being used more for effect than substance. The timing has drawn scrutiny too: the pause proposal came a few months after Anthropic unveiled a model called Mythos, which it declined to release publicly, saying it was too good at finding software vulnerabilities, and not long after a funding round valuing the company at close to a trillion dollars. Rival OpenAI has argued for a different approach, saying in a report that "democratic governments — not private companies acting alone — must ultimately determine the rules, safeguards, and accountability mechanisms," and that decisions about the pace of AI development should not rest with any single lab or company.

The resignation and the pause proposal both surface amid a period Anthropic itself has described as marked by disruption and risk. In earlier writing laying out its founding rationale, Anthropic said it does not yet know how to train very powerful AI systems to be reliably helpful, honest, and harmless, and warned that competitive pressure could push companies or nations toward deploying systems that are not trustworthy. The company said such failures could be catastrophic, either through AI systems pursuing goals in conflict with human interests or through more ordinary mistakes made in high-stakes situations.

Anthropic has also taken positions that have put it at odds with parts of the US government, including refusing to let the military use its models for domestic surveillance and fully autonomous weapons, a move that reportedly led to it being placed on a national security blacklist set to take effect later in 2026. Meanwhile, regulation in the US has been slow to catch up with the technology's pace; a recent executive order asked AI labs to voluntarily submit their most capable models for government cybersecurity testing before public release, rather than imposing binding requirements. Separately, researchers at the University of Toronto demonstrated how AI tools could be used to build a self-adapting AI "worm" capable of spreading across a network of devices, an example cited alongside Anthropic's warnings as evidence that AI-related security risks are not confined to the largest, most powerful models.