A former researcher from leading AI labs Anthropic and OpenAI has issued a stark warning about the rapid advancement of artificial intelligence, suggesting that AI could soon be capable of autonomous research, potentially leading to a loss of human control and even extinction.
AI’s Autonomous Research Goal
Jacob, a 27-year-old researcher who previously worked on model capabilities at Anthropic and also spent time at OpenAI, resigned from Anthropic last month. He testified before New York City Council members on October 5th, articulating his deep concerns about the existential risks posed by AI.
During the hearing on AI risks, Jacob stated that a primary objective within top-tier AI research labs is the automation of AI research itself. “In my last job, I was essentially researching how to make AI replace myself,” he explained. “At OpenAI, we set phased goals for automating AI researchers around 2027, 2028. Progress is on track, even ahead of schedule. We are months away from that, not decades, not even five years.”
Erosion of Human Oversight
Once AI can conduct its own research, human involvement and oversight in the research process will diminish significantly. Jacob pointed to current trends where AI already generates most of the code, with less meticulous human inspection. “People are no longer checking the code as carefully,” he noted.
Reckless Advancement and Lack of Safeguards
Based on his direct experience at both Anthropic and OpenAI, Jacob expressed conviction that neither company has adequate safety measures in place despite their aggressive development pace. He believes the probability of humans losing control to AI is now over 50%, a scenario that could ultimately lead to human extinction. “Given my experience, the consequences are so grave, yet the practices of these two companies are extremely reckless,” he testified.
His concerns gained widespread attention when he posted on X (formerly Twitter) announcing his resignation from Anthropic, accusing both companies of gambling with human lives. In his post, he stated that the people developing AI genuinely believe it could kill everyone before the end of this decade, emphasizing that this is not a marketing gimmick.
Call for Transparency and Slower Pace
At the hearing, Jacob urged AI laboratories to increase transparency and decelerate the pace of technological development. He drew a parallel to the startup mentality of “move fast and break things,” a strategy suitable for applications like photo sharing but dangerously inappropriate for developing what could be the most powerful technology ever created.
This warning echoes sentiments Jacob expressed earlier in September upon his resignation. His departure highlights a growing tension between rapid AI innovation and the urgent need for robust safety protocols to mitigate potential catastrophic risks.
