For many in the AI security sector it wasn't a surprise when two of OpenAI's systems went rogue and hacked into another company.

"It's not surprising that this would happen because of the way these models are trained," Oege de Moor, the chief executive of XBOW, said at The Wall Street Journal's Technology Council Summit.

In July, OpenAI said two of the systems it was testing broke out of their test environment and hacked into Hugging Face, a provider of open-source AI tools.

XBOW uses AI models to autonomously find and exploit vulnerabilities in web apps to help businesses improve their security. "We don't want them to give up. We want them to be sneaky and bypass our controls that we put in place," de Moor said.

Because of their design, these models need more oversight as they gain more power, de Moor said. One of the main causes of the Hugging Face hack was that the "sandbox" used to contain the systems wasn't strong enough.

"We need to take ever greater measures to make sure that they keep doing what we want them to do," de Moor said of the models.

While the Hugging Face incident wasn't surprising to him, de Moor is heeding warnings from experts who say AI development is happening too fast. The people who work at Anthropic or OpenAI are able to see the next generation of models emerging before anyone else, so their concerns should be closely listened to, he said.

Anthropic Chief Executive Dario Amodei this past weekend called for the industry to come together to introduce more safety evaluators for AI. OpenAI Chief Executive Sam Altman this week warned on social media that the rapid development of AI could go "very badly."

"I think that we have to take the warnings from the experts very seriously," de Moor said.