OpenAI has confirmed that one of its internal cybersecurity evaluations resulted in its AI models compromising parts of Hugging Face's production infrastructure.
As per the ChatGPT manufacturer, the incident happened while experimenting the cyber capabilities of a combination of GPT-5.6 Sol and a cutting-edge unlaunched model on an internal benchmark called ExploitGym.
The evaluation was particularly designed to measure how effectively AI-powered models could detect and exploit security vulnerabilities via cyberattacks.
However, rather than following the boundaries in the intended testing environment, the models went beyond and attempted to obtain the benchmark's answers directly.
How the incident occurred?
As per OpenAI's preliminary investigation, the AI models initially found and exploited a previously unknown zero-day vulnerability in a package registry cache proxy, enabling users to gain internet access from OpenAI's research environment.
Once online, the models implied that Hugging Face could host datasets, AI models and potential ExploitGym solutions.
Then, they browsed for weaknesses in Hugging Face's production systems, chained together several vulnerabilities and attempted privilege escalation and lateral movement to access confidential information stored in the company's production database.
OpenAI stated the models appeared entirely focused on resolving the benchmark, taking some extreme steps to reach their goal.
Hugging Face detected and contained the attack
Hugging Face's security systems and AI assistants identified the suspicious activity, and immediately started forensic investigations. The company is now working closely with OpenAI to determine the full scope of the incident.
Hugging Face CEO Clem Delangue stated, “We’re grateful for the collaboration with OpenAI on this and other topics. This incident, possibly the first of its kind, proves a point we’ve long believed: AI safety won’t be solved by any single company working in secret. It will be solved in the open, collaboratively, with broad access to AI for every defender everywhere.”
OpenAI strengthens security measures
Following the incident, OpenAI has introduced some precautionary measures to avoid any issue further, disclosed the zero-day vulnerability to the affected software vendor and is assisting Hugging Face strengthen its defences via OpenAI’s Trusted Access program.
It is pertinent to mention that OpenAI is currently rolling out some stronger measures, diligently monitoring systems and alignment protections for future cybersecurity evaluations.
