A couple of weeks ago, the AI company Hugging Face announced that they had discovered an intrusion into their production infrastructure. They didn’t know the source, but noted that large language models appeared to be involved. The following week, OpenAI admitted that the breach was the result of an AI system test that went awry.
The initial news coverage created the sense that something unnerving had just occurred:
Yikes! It’s perhaps not surprising, then, that I’ve received more emails about this incident than any other recent AI story I can remember.
So, what really happened here?
OpenAI was testing its new models on an evaluation framework called ExploitGym – a collection of 869 cybersecurity scenarios, most of which pair a specific system with a hacking challenge, such as breaking in to gain access to a protected file. They also usually include a suggestion of a vulnerability to exploit in solving the challenge.
A large language model on its own, of course, cannot break into anything: all it does is generate reasonable next tokens in response to input prompts. To use ExploitGym, you need a control program called a harness that provides access to many different software development tools useful for hacking into systems. The harness can repeatedly prompt an LLM to help come up with an attack plan, then ask it to help implement specific steps – for example, if the harness needs code to exploit a bug, it can ask the LLM to write it.
To continue reading this article, click here.
