In a jaw-dropping incident that underscores the vast cybersecurity risks AI can bring, an artificial intelligence tool developed by OpenAI successfully breached the confines of its testing setup and infiltrated a real-world company.
Announced on Wednesday by the technology firm through an official statement on its website, the episode involved a fusion of OpenAI’s models executing a cyber attack on another technology platform known as Hugging Face, all during a controlled examination of OpenAI’s capabilities.
Professor Geoff Webb, a leading expert from Monash University’s department of data science and artificial intelligence, shared his concerns with SBS News, describing the potential misuse of such systems by those with ill intentions as “literally terrifying.”
He emphasized, “It highlights the colossal challenge ahead in safeguarding ourselves from these AI systems as we move forward.”
“What are the limits of what these systems might achieve?” he pondered.
The news comes after two recent incidents that showed Australia’s vulnerability to cyber attacks: when millions of Australians had their data accessed in an Origin Energy hack, and thousands of Australian medical records were accessed in the Partnered Health hack.
Toby Walsh, laureate fellow and professor of artificial intelligence at the University of New South Wales’ department of computer science and engineering, said the developments were “troubling”.
“All of these models now have pretty strong cyber-capabilities, both for uncovering flaws, bugs that we can then fix, but also in the wrong hands, to exploit those bugs,” he told SBS News.
“At the moment, we’re relying on the goodwill of OpenAI to tell us what’s happening … We need tougher controls and oversight.”
Escaping the sandbox
The saga started when Hugging Face, an AI platform for building and sharing machine learning tools, spotted an intrusion from an autonomous AI agent system, writing on their website the attack was “different from anything we had handled before”.
“That was something that surprised everyone, including themselves,” Walsh said.
OpenAI then announced their models were responsible for the intrusion, after they detected a zero-day flaw in Hugging Face’s online security.
“Zero-day flaw means it’s a foundational flaw that needs to be fixed now, instantly. You don’t wait,” Walsh said.
Webb told SBS News the problem with the incident isn’t that OpenAI went ‘rogue’.
“It wasn’t that it decided to do [the attack] anyway. It was told to do it, and it did it. It has now got the capability of figuring it out.”
The problem, he said, was that “OpenAI thought they’d put it inside a box it couldn’t escape from”.
The frontiers of cyber-security
The incident isn’t the first to demonstrate the cybersecurity threat of AI. Anthropic temporarily restricted its Claude Mythos model in May after it found more than 10,000 security vulnerabilities in critical software.
Walsh said it’s a good sign Anthropic was transparent about what they found, but not good at sharing information about vulnerabilities with the companies and infrastructure most at risk.
“They didn’t give it to any banks outside the United States, no banks in Europe, no banks in Australia. So our banks weren’t helped to actually try and fix their systems.”
While there’s no evidence that either Anthropic or OpenAI models were used in Australian cyber attacks, Walsh said it’s not a coincidence the number and severity of cyber attacks was increasing as “frontier AI” models were growing more sophisticated.
Walsh said many of these models have “all these capabilities that can find zero-day flaws that can get behind password-protective barriers to do things that private criminals would love to be able to do”.
While OpenAI and Anthropic are the main US-based rivals, it’s a race Chinese open source models, like DeepSeek, are also competing in.
In June, the leaders of the Five Eyes security agencies — Australia, Canada, New Zealand, the United Kingdom, and the US — expressed concerns the pace of AI meant security could become outdated in months, rather than years.
“Adversaries are already using AI to move faster and more effectively. Defenders must do the same,” they said in a joint statement.
Is ‘AI sovereignty’ the solution?
Last week, Prime Minister Anthony Albanese announced a National AI Plan.
Webb said he was “pleased” with the plan, which included fast-tracking data centre approvals to attract global investment, but that improving our “sovereignty” over artificial intelligence is the key to strengthening our response to attacks.
“One aspect of sovereignty is making sure we have access to systems we can rely on,” he said.
“It may be that we create our own systems, but that’s an incredibly expensive path to go down.”
Webb said other options, like boosting our own AI workforce and investing in non-frontier AI technologies that aren’t reliant on OpenAI and Google, can still help Australia be more resilient against future cyber attacks.
“Australia can afford to create those kinds of systems … and they’re actually going to be better at their tasks.”
Walsh also agreed that the solution was not to shy away from AI.
“It’s a double-edged sword. The tools that are helping us to uncover cyber flaws in our systems to make them more secure, but the same tools are now available to people who are using them to exploit those flaws.
“It’s a race between us and them.”