Anthropic CEO Dario Amodei and others are warning that AI models may be able to build on themselves, without the help of humans. It's a concept known as recursive self-improvement.

A process known as recursive self-improvement could "quickly yield AIs that are intelligent beyond our comprehension," believes Alex Turner, former Google DeepMind researcher who recently resigned over AI safety concerns.

As AI leaders begin to sound the alarm on possible risks, a major factor fueling their concerns is the ability for AI models to build the next generation of AI systems - without humans.

There's evidence that that process, known as recursive self-improvement, is already happening in pockets of some AI labs. It's hard to say how pervasive the phenomenon is, but the examples available to AI leaders so far have them concerned.

Recursive self-improvement was a major focus of an essay published Saturday by Dario Amodei, Anthropic's CEO, which called for leading AI companies to slow the pace of development.

It's "starting to happen across the industry," including at Anthropic, Amodei wrote of recursive self-improvement.

"Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all," he added.

"Dario is right," Elon Musk wrote on X, sharing Amodei's piece, which highlighted "AI's growing ability to build the next generation of AI."

"I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we've had at OpenAI in recent weeks," OpenAI CEO Sam Altman wrote on X on Saturday.

For some prominent AI experts, the Hugging Face incident raised alarm bells about the potential for recursive self-improvement. In July, Open AI models circumvented human controls placed on them and hacked into Hugging Face, a repository for open-source AI models.

"Today's rate of AI progress is staggeringly fast," according to Alex Turner, a former Google DeepMind researcher who resigned so he could document Google's "broken promises" on a number of ethical issues. He weighed in on the matter in an op-ed for the Guardian on Monday.

"Fast progress today means even faster progress tomorrow, driven by tomorrow's even smarter AIs. The progress would enter a feedback loop called 'recursive self-improvement.'" He cited the Hugging Face incident as an example.

He wrote that RSI could "quickly yield AIs that are intelligent beyond our comprehension."

Jakub Pachocki, chief scientist at OpenAI, wrote in a blog earlier this month that based on "internal results" he expects the speed of the development of AI systems could be "sustained into recursive self-improvement."

Pachocki wrote that in mid-2023 he saw the "first results" that indicated one day pretrained AI models would have the capability to form their own chains of thought.

"Szymon and I spent that night at the office, thinking not about the incredible benchmark numbers, products, or scientific results that this technology will deliver - but rather, trying to process the sobering fact we will actually see machines meaningfully smarter than ourselves in our lifetime," he wrote, referring to Szymon Sidor, a researcher at OpenAI.

OpenAI is currently hiring for a research scientists and engineers on the company's Recursive Self-Improvement team.

Sobering milestone may not be as close as some think

Concerns about RSI in the AI community are not universally shared.

James Zou, who leads the Artificial Intelligence for Science Lab at Stanford University, told MarketWatch that it's unclear whether any AI labs have actually reached this loosely defined milestone. There's no "widely agreed-upon threshold or criterion" to indicate when RSI has been reached, he said. He defines RSI as when "AI systems autonomously improve the capability of future AI systems."

For instance, an AI model could "create a better training algorithm and then apply this method to create a new model that is even better at coding or reasoning."

He said RSI is "not cause for panic," though important to monitor as it could "accelerate AI development and lead to more autonomous or unexpected AI capabilities."

See also: Buy these stocks to ride the next big AI wave - connectivity, analyst says

Armando Solar-Lezamathe, associate director of the Computer Science and Artificial Intelligence Laboratory at MIT, told MarketWatch that definitions in the artificial-intelligence field, like artificial general intelligence and RSI, are sometimes a bit "fuzzy."

"There's a good argument to be made that they're kind of marketing terms" that can be redefined by AI labs, he said.

He explained that the concepts of AGI and RSI rely on the assumption that intelligence operates on a "linear" scale, such that once a machine reaches a level where it's smarter than a human, it doesn't need humans to keep improving itself. He said the assumption can be problematic, and that intelligence is not necessarily so linear.

"Everyone agrees that [RSI] involves AI systems leading the process of building the next generation of AI systems, but there is a lot of debate around at what point you can claim you have RSI and what are the implications," he said.

-Hannah Pedone