A 27-year-old artificial intelligence researcher, Jacob Cockson, has warned that increasingly powerful AI systems could pose an existential threat to humanity and, under a worst-case scenario, could potentially contribute to the extinction of humans before the end of the decade.
Cockson, who spent around three years working on AI training research at OpenAI and Anthropic, issued the warning after resigning. He argued that both companies are engaged in a race to develop increasingly powerful and potentially self-improving AI systems, raising concerns about whether safety measures are keeping pace with technological progress.
According to Cockson, several senior executives and researchers in the AI industry privately acknowledge the possibility that advanced AI could become an existential threat. However, competition between companies and the fear of falling behind rivals continues to drive rapid development.
He warned that future superintelligent systems could potentially outperform humans in areas including cyber operations and could gain access to real-world resources and forms of power at a speed and scale humans may struggle to control.
Cockson called on AI researchers to demand much stronger safeguards before developing systems with extreme capabilities. He also suggested that difficult measures, including temporarily slowing or restricting capability increases, may need to be considered to prevent a dangerous global race.
Anthropic's alignment research chief, Evan Hubinger, has also expressed serious concerns about the possibility of AI causing human extinction. Hubinger said he personally considers the probability of such an outcome during the next decade to be above 10 percent.
He acknowledged that Anthropic is working extensively on AI safety but said the problem of aligning superintelligent AI with human objectives has not yet been completely solved. The company, he indicated, does not currently have a complete roadmap for addressing the challenge.
Experts distinguish between the risks posed by today's AI systems and those associated with a future superintelligence capable of repeatedly improving itself. The greater concern is what could happen if AI systems begin advancing their capabilities faster than humans can understand, regulate or control them.
