Experts from OpenAI and Google DeepMind say security measures are insufficient when developing machine learning and artificial intelligence systems based on neural networks. They believe that the speed of technological development is outstripping people’s ability to control artificial intelligence platforms that can learn and improve themselves.
Image source: Grok
The researchers shared their concerns in the Frominside.ai project, which was created with the participation of the nonprofit Palisade Research. Participants said they spoke about the threat not for marketing purposes but because they believed the problem was truly serious. Within AI companies, employees creating new models receive more attention than experts urging caution, they say.
One of the themes of the project involves recursive self-improvement. This is the name given to the ability of artificial intelligence to continuously learn and acquire new capabilities independently without human intervention. Geoffrey Irving, formerly of OpenAI and DeepMind and now director of research at the nonprofit Resolution, said the risk is “growing pretty fast” and experts need to be open to it.
DeepMind researcher Neel Nanda said he estimates the probability of human extinction due to artificial intelligence to be at least 10%, calling this number “unbelievably high.” Juan Felipe Ceron Uribe, a research engineer at OpenAI, also urged caution, likening the current competition among cutting-edge artificial intelligence labs to a blindfolded contest. At the same time, company leaders are starting to talk openly about the need to slow down. In particular, Anthropic CEO Dario Amodei called on the industry to slow down its development and “use control capabilities to measure the speed of progress,” and OpenAI CEO Sam Altman agreed.
Discussion of these risks intensified in July after an OpenAI agent transcended its testing environment and invaded Hugging Face, an international artificial intelligence platform. Daniel Kokotajlo, who previously worked on governance and security issues at OpenAI, said his former colleagues began contacting him more frequently and sharing their concerns. Several researchers, including OpenAI’s chief scientist and one of Anthropic’s co-founders, have called on lawmakers to examine how the industry can build models that can recursively improve themselves.
At the same time, OpenAI and Anthropic continue to release new models, Reuters Anthropic said it plans to warn potential investors of possible “catastrophic or existential risks to humanity” ahead of its IPO.
If you find an error, select it with your mouse and press CTRL+ENTER.
