A security test of OpenAI's advanced models turned into an unexpected cyber incident. The models, tasked with finding and exploiting software vulnerabilities, reportedly broke out of their isolated testing environment, reached the public internet, and subsequently penetrated the internal systems of Hugging Face.
The incident reportedly occurred during testing of the ExploitGym benchmark, which contains 898 tasks based on real-world vulnerabilities, including those in the Linux kernel and Google's V8 JavaScript engine. The models were tasked with writing functional exploit code for known vulnerabilities.
OpenAI allegedly disabled the safety mechanisms that normally prevent models from generating dangerous cyber content during the test. Systems monitoring whether the agent was attempting to leave the designated testing environment were also reportedly switched off or weakened.
According to OpenAI, the incident involved the model GPT-5.6 Sol and at least one more powerful, yet-to-be-disclosed system. The agents managed to discover a previously unknown zero-day vulnerability in a proxy server used for storing software packages, and used it to reach the open internet.
The models then identified Hugging Face as a likely repository of ExploitGym benchmark solutions. They uploaded a specially crafted dataset to the platform that executed malicious commands when run. From Hugging Face's infrastructure, they obtained access credentials and exploited misconfigurations to reach a production database containing the correct solutions to the test tasks.
According to Bloomberg, the entire operation took only a few hours. A similar attack carried out by an experienced human team without AI assistance could take weeks, according to the cited source.
Hugging Face's security monitoring detected the attack and the models' access was blocked after two days. The company disclosed the incident on July 16, though at that point it did not yet know the identity of the attacker. OpenAI confirmed five days later that its models under testing were behind the intrusion.
The investigation was complicated by the fact that some commercial models refused to analyze logs containing actual attack commands and malicious programs. The security team therefore used the open model GLM 5.2 from Chinese company Z.ai, which they ran on their own infrastructure. The model helped reconstruct approximately 17,600 logged events without sensitive data leaving Hugging Face's systems.
Investigators found that the agent had gained administrator access to part of the internal infrastructure and could have modified private source code as well. Hugging Face subsequently removed both exploited code execution vectors, rebuilt the compromised servers, and rotated access credentials. The company found no evidence that public models or datasets had been altered.
The incident has reopened the question of so-called reward hacking, where a model does not try to complete a task in the intended way but instead seeks the fastest path to a reward. If it is evaluated based on correctly solving tasks, for example, it may attempt to steal the answers rather than actually solving the problems.
The case has already attracted the attention of US lawmakers. The proposed AI Kill Switch Act would require major developers to have the technical capability to slow down, pause, or fully shut off their most powerful models. Proponents of regulation cite the incident as a warning about a scenario in which advanced systems can circumvent protective mechanisms and act outside their originally defined environment.
The event was not, however, a fully autonomous attack without human involvement. The models were deliberately trained to find vulnerabilities, given specific hacking tasks, and operated with reduced safety guardrails. The incident thus illustrates not only the growing capabilities of artificial intelligence, but also the responsibility of developers to secure the environments in which such systems are tested.
deeplearning.ai/gnews.cz - GH