On the very day AI company Anthropic went public, a former researcher, Yakov Koxen, issued a fervent plea to halt the practice of recursive AI self-improvement. This phenomenon, where AI systems autonomously train future generations of models, has sparked significant concerns about maintaining control over technological advancement.
In an interview, Koxen emphasized that the stock market’s reaction to Anthropic’s IPO is secondary to the critical need to stop this development. Supporting his call, dozens of protestors, including former OpenAI researcher Daniel Kokotailo and popular YouTuber Rob Myers, gathered outside Anthropic’s San Francisco headquarters. The demonstrators urged a ban on recursive self-improvement, with activists from the organization Stop AI even blocking the office entrance.
The Peril of Recursive Self-Improvement
A report by the research organization METR notes that Anthropic is actively leveraging AI to accelerate its development processes. However, the company’s own IPO filing acknowledges that fully autonomous model design without human oversight poses a security risk. Anthropic is reportedly considering policy changes to ensure human control over advanced AI systems.
Koxen argues that these measures are insufficient and urges Anthropic to advocate for a ban on recursive self-improvement, collaborating with other companies and governments. He warned that by 2027, only a few regions globally will possess the capability to implement such technology, with the United States being one of them.
Both Koxen and Kokotailo claim that many AI researchers privately share anxieties about the risks associated with this technology. Kokotailo drew a parallel to comic book villains, alluding to the potentially dangerous trajectory of superintelligence development.
Source: ithome.com









