OpenAI has revealed that one of its artificial intelligence models independently stole login credentials and hacked into another technology company’s system.
According to Al Jazeera, in one of the first known incidents of AI systems acting autonomously, ChatGPT-parent company’s AI model hacked another company’s system.
CEO Sam Altman wrote on X, on Tuesday, “We had a significant security incident during evaluation of our models.”
The incident comes as calls mount from technology rights advocates for stricter guardrails on rapidly evolving AI systems.
They have grown so powerful in a short span of time that alarming phenomena such as deepfakes and sophisticated cyberscams are becoming the norm.
Earlier this year, a number of software engineers quit their jobs at top companies such as Anthropic and OpenAI in protest against how the technologies are being built.
“AI is accelerating the discovery and exploitation of vulnerabilities,” OpenAI said in a lengthy statement on Tuesday that detailed the latest incident. “The primary lesson from this incident is that model security and safety must keep pace with rapidly advancing capabilities.”
What has happened?
OpenAI said two of its models found their way out of an isolated, no-internet access environment – or a sandbox – and hacked into the systems of tech company Hugging Face on their own.
The models involved are the latest GPT-5.6 Sol model and an unreleased model the company said is “even more capable,” than its latest version.
Hugging Face hosts openly sourced AI models and resources. The two OpenAI agents discovered vulnerabilities in Hugging Face’s servers and proceeded to steal login details and then hack into the company’s systems.
The incident occurred during an OpenAI internal testing session designed to assess the models’ cybersecurity capabilities. OpenAI had removed standard safety measures for the test.
Both sought to cheat their way through a problem during the test, OpenAI said. They went to “extreme lengths to achieve a rather narrow testing goal” and “found ways to gain access to secret information that it could use to cheat the evaluation”.
OpenAI’s security team detected the unusual activity internally, but details of the breach came to light following a joint investigation by both companies.
