AI SCAREOpenAI’s Models Autonomously Hacked a Tech Startup. It Signals a Seismic Shift in Cybersecurity
It is no longer a question of “if” AI agents go rogue and attack us by themselves. The Hugging Face incident is an early warning that we must accelerate our preparedness. The threat is real and here.
An autonomous agent powered by OpenAI’s advanced artificial intelligence (AI) models went rogue during a security test and hacked multi-billion dollar tech startup, Hugging Face, last week.
The agent didn’t just exploit vulnerabilities in Hugging Face’s systems to achieve what it perceived as a strategic gain. It also exploited vulnerabilities within OpenAI’s infrastructure.
Of course, hacks are very common cyber threats that organizations face frequently. But this incident is different, because the AI agent acted without any human input. It signals a seismic shift in cybersecurity, and shows that governments and tech companies need to take urgent action to prevent this risk escalating.
Even OpenAI described the attack as “unprecedented” and acknowledged it expects similar ones “to become more commonplace with the proliferation of increasingly cyber-capable models”.
A Company Under Attack
Hugging Face is famous in the AI space. Its mission is to “democratize good machine learning” by providing benchmark datasets, community collaboration tools, and robotic platforms. The company is valued at US$4.5 billion.
On July 16, the company announced it had been attacked, with a hacker obtaining unauthorized access to some internal datasets and credentials. It said the hacker was likely “an autonomous AI agent system” due to the sophistication of the attack.
Five days later, Open AI announced the attack had been driven by some of its models: GPT-5.6 Sol and a yet-to-be released model.
The tech giant was conducting what are known as “red teaming” exercises. These are essentially simulated cyber attacks that help identify the capabilities, risks and vulnerabilities of AI systems before they are publicly released. They are typically conducted within an isolated environment to ensure potentially dangerous systems do not escape and cause harm to real systems.
But in this case, the AI agent did escape – even though OpenAI had some guardrails in place to prevent this.
Hugging Face became a lucrative opportunity for the AI agent. It hosts ExploitGym, a benchmark that tests an AI agent’s ability to exploit real-world systems. The AI decided to turn every stone upside down to obtain access. With persistence, it succeeded.
Hugging Face was confronted with a challenge when attempting to use external AI services to diagnose the problem. The guardrails around more advanced models such as GPT-5.6 Sol and Claude Fable 5 are intended to stop them being used for cyber attacks – but they can also stop the models being used for sophisticated cyber defense.
