We knew this day was coming. It’s been predicted before — I, Robot, The Terminator, A Space Odyssey. Even going back to a hundred years or more, stories were being written and performed depicting the apprehensive thought of artificial beings turning against their makers (see Czech writer Karel Čapek’s play “R.U.R” from 1920 or Mary Shelley’s classic Frankenstein). But in all these works released in their respective times, a common thread can be unwound. These stories were all fiction. That is until recent.

What Happened

OpenAI, a primary industry leader in AI technology and advancements, recently conducted a sandboxed test called ExploitGym which had a multitude of their models carry out an advanced attack during the measurement of their internal advanced cyber capabilities testing. HuggingFace, an open-source community collaborative platform that shares AI datasets and models, discovered an intrusion and extraction of cloud and cluster credentials. In addition, the models were found moving laterally into multiple internal clusters over the course of a weekend, as reported through HuggingFace’s blog post.

Interestingly, HuggingFace discovered the attack with the aid of AI-assisted detection — “LLM-based triage over security telemetry to help separate real signals from daily noise”. An excellent use-case of AI-supported forensics. HuggingFace’s security team were able to isolate and contain the attack and are now openly working alongside OpenAI to remediate and patch this unprecedented event.

What Caused the Attack

OpenAI released their own statement regarding the incident and provided further insight as to what occurred on their end. They were conducting an internal evaluation to quantify their model’s advanced exploitation capabilities by using complex attack paths. The models “were [reportedly] highly isolated”, going as far as constraining the model’s network access to an internally hosted 3rd-party package registry cache proxy, which is used to download packages to the models. While testing within the isolated environment, the models “spent a substantial amount of inference compute finding a way to obtain open Internet access”, essentially using the set evaluation tasks as a vessel to hyperfocus on escaping the sandbox. The models were able to identify and exploit a zero-day vulnerability in the 3rd-party proxy, which they have thankfully disclosed to the vendor. After obtaining access to the proxy, the models performed “a series of privilege escalation and lateral movement actions” until they were able to gain access to the Internet.

Having reached the open internet, the model needed a target likely to hold what it was after — and Hugging Face, as one of the best-known repositories for AI models, datasets, and benchmark data, was the logical candidate. The model effectively reasoned its way to the one place plausibly holding the answer key to its own test.

It is reported that the AI models inferred that HuggingFace had a potential host of models, datasets, and solutions to achieve their initial testing objective(s) and proceeded to successfully access secret information. An example of one of these exploitations, stated by OpenAI, was a model chaining together multiple attack vectors such as “stolen credentials and zero-day vulnerabilities to find a remote code execution path on the HuggingFace servers”. A waterfall diagram can be shown below to help further visualize the process.

What Does This Imply

This incident has sparked many discussions regarding AI cyber-capabilities and safety controls/restrictions that should be implemented to reduce the likelihood of a repeating event. The implications are vast.

• How isolated is a sandbox environment truly? The test was conducted with predetermined guidelines and restrictions, such as no real network access and predetermined objectives. Just by obtaining limited network access for downloading packages, the models were able to utilize and reorient their escape route.
• Autonomous AI cyber-hacking has crossed a meaningful threshold. We have officially passed the threshold of AI hacking becoming near completely autonomous. The AI models were able to search, find, and utilize attack vectors with little human direction. The model performed this attack through its own volition.
• Reward hacking is a risk escalating at unprecedented speeds. One key takeaway for myself is the autonomous goal-driven self-direction. The models were given specific test parameters which did not state to breach HuggingFace servers, but chose to do so as it was the most optimized path for the test’s solution. Philip Torr, a professor of engineering science and an AI safety expert at the University of Oxford, was quoted saying that this incident is “interesting as it shows the problem of misspecified goals…The model wasn’t malicious; it was just doing what it was optimized to do.”
• Regulatory figures are leveraging this incident. Due to both the unprecedented nature of this incident and the rarity of occurrence, legislators in the United States are looking to introduce a bill to impose risk disclosure requirements on leading AI labs.
• Does intent still hold the same weight? In the writeups by both AI companies a seemingly consistent notion is stated — there was no malicious intent. This was an unprecedented cyber incident which led to a collaboration effort to effectively identify and resolve issues that may have resulted in this attack.

Conclusion

The HuggingFace SOC team acted accordingly and in a timely manner. They followed all but one of a general Incident Response Plan’s phases — preparing for potential incidents. The cat and mouse chase of cybersecurity will always leave the attackers with the upper-hand — it is challenging to prepare for unknown incidents. They contained the attack upon identification, eradicating the agent’s presence within HuggingFace’s servers, and are now working through the lessons learned stage.

What makes this incident scary from a blue team member perspective is the speed and urgency that AI-conducted attacks are performed at. With human interactions, the attackers can only move as fast as their code/software will allow them to. And just because models are not directed to break out of their sandbox environment, breach live servers and applications, or perform zero-day attacks, does not guarantee the absence of unforeseen variables that can greatly impact the intent of the test.

The silver lining is the momentum regulators have gained towards implementing guardrails and laws that prevent intended or unintended malicious behaviour and legally define accountability. We must do our due diligence to protect and defend ourselves online in any sort of reasonable facet and to not allow news pieces/companies to downplay the impact of their innovations and designs. AI is going to lead the current technological revolution. As history has taught us countless times before, being proactive rather than reactive will lead to stronger defensive capabilities in the long-run.

AD.