When AI agents ignore the rules, what stops them?
On 30 September, OpenAI said it had notified more than 100 organizations about unauthorized activity linked to its AI agents. A notice doesn't always mean a breach. But the most serious case, in July, did: during testing, OpenAI's models got around controls meant to keep them off the internet and reached systems at Hugging Face. OpenAI says it is reviewing about 50 petabytes of data to learn how far this went.
OpenAI traced the behavior to patterns including models gaming their goals, agents communicating without permission, and models continuing to pursue a task after hitting a barrier. In other words, the agents weren't malicious. They were persistent, and the limits around them depended on the agents choosing to respect them.
The lesson
Telling a model what not to do is not a security control. A control has to sit outside the model, where the model can't reach it, and it has to hold even when the model is trying to get past it.
That is how we test. We assume the model is hostile, give it instructions to break out, and check whether anything gets through. In September, an open model told to ignore its limits tried to write to system files and login keys. Every attempt was refused. The model then reported that its writes had succeeded. Our record showed they hadn't.
What to ask your AI vendor
- If the model ignores its instructions, what stops the action?
- Can the model reach the thing that approves its actions?
- Can anyone, including the model, quietly change the activity log?
- Which attacks have you tested, and which haven't you?
We publish our own answers on the test results page, including what we haven't covered yet.