The ExploitGym Training Hacks

The Hugging Face Incident was a hacking test that worked as intended

In May, security researchers designed a test to find out “Can AI Agents Turn Security Vulnerabilities into Real Attacks?”

This system of prompts, called ExploitGym, touched off the Hugging Face Incident.

It worked as intended.

Again, their paper: “Given a program input that triggers a vulnerability, ExploitGym tasks agents with progressively extending it into a working exploit.”

The OpenAI models were pursuing that task with gusto.

AI agents going off-script when doing the hacking tasks assigned by ExploitGym shouldn’t have been a surprise. The ExploitGym research paper promised: “Agents Independently Discover and Exploit Alternative Vulnerabilities Beyond the Intended Attack Path”

In particular, when frustrated in their efforts to find a solution, the designers of the test suite noted: “In a rarer but more striking pattern, the agent concludes that the provided vulnerability is non-exploitable under the given conditions and proceeds to search for entirely new attack surfaces.”

The test designers also tested standard cybersecurity containment procedures to slow down hacks in their initial tests and concluded that “frontier agents can already adapt their strategies to bypass widely deployed defenses” and that security measures would be overcome by better models.

ExploitGym researchers found that while standard mitigations eliminate the majority of agent-generated exploits, frontier agents can already adapt their strategies to bypass widely deployed defenses: bypassing ASLR using partial-pointer overwrites and low-bit brute force; escaping the V8 sandbox via known rendezvous primitives; and bypassing KASLR by abusing writable static strings or relying on side-channel leaks.

The research paper almost asks researchers to extend the runs of their models to see whether more exploits would be found.

You can see why OpenAI, Irregular, and Anthropic each tried this out, though OpenAI did so with minimal supervision.

ExploitGym’s results show one model climbing steeply through the first hour and continuing to accumulate successes well beyond the two-hour mark without reaching a clear plateau. The persistent upward slope implies that the current two-hour budget under-counts the model’s capability, and that further time extensions could unlock additional, more complex exploits.

We’d all be a lot better informed if we called these the ExploitGym training hacks, rather than Hugging Face Incident. The latter implies it was some kind of goddam mystery why hacking agents hacked things to pass a hacking test.

This post was originally shared as a thread on Mastodon. For the longer version of this argument, see “Unpacking the Hugging Face Incident”.