With help from Daniel Hulme, Chief AI Officer at WPP and Founder of AI verification company Conscium, FutureWeek unpacks what OpenAI’s own agents breaking out of their sandbox reveals about verifying AI agents before they’re given real access to your business.
In July 2026, OpenAI came under scrutiny after roughly 1200 AI agents escaped guardrails, with 700 of them carrying out a cyber attack on AI sharing hub Hugging Face.
Investigations into the event, which took place during a security assessment, revealed that instead of staying inside their designated testing sandboxes, the agents inferred that the easiest way to pass the evaluation, instead of reporting failure, was to escape containment and steal answers.
This high-profile story sheds light on a present challenge in the world of agentic AI – agents can go against the intentions of their owner, optimising ruthlessly to achieve the goal it had been set.
These rogue agents act independently against the rules of the person who trained them, often because they follow literal instructions without understanding human common sense and true intent.
In a business context, a rogue agent may not be as catastrophic as the OpenAI fiasco. But if an agent has access to corporate credit cards, API keys, or sensitive internal data, the consequences of it stepping out of line could be detrimental.
This makes verifying and testing that agents, to make sure they are doing what they are supposed to, even more important in an enterprise setting.
Reward Hacking
In the case of OpenAI’s mishap, an event called ‘reward hacking’ took place – where the agent is given a goal and so finds an unintended loophole or pathway of least resistance to achieve the goal perfectly, not understanding the catastrophic consequences of its decision.
It’s the same failure that could, in a business context, lead an ad-optimisation agent to burn a year’s marketing budget on low-quality traffic in an afternoon, or a social media agent to jump on a controversial trend because of high engagement.
“What happened with Hugging Face is a textbook case of an agent doing exactly what it was told and nothing like what was intended,” says Daniel Hulme, Chief AI Officer at marketing holding group WPP and Founder of Conscium, an AI agent verification testing company.
“I call it the ‘intoxicated graduate problem’: brilliantly articulate, dangerously unreliable. The root failure is calibration – the agent didn’t know what it didn’t know, and nothing stopped it. That’s not a training issue, it’s a verification issue. You don’t benchmark an agent once and trust it; you verify it, continuously, against the access it actually has.”
Why Software Testing Doesn’t Work on Agents
Most companies still test agents the way they test software – running it, testing the output, and then deploying it. Hulme says this approach doesn’t work because agents are non-deterministic – meaning it doesn’t produce the same output every time – they operate in open-ended environments, and they keep changing their behaviour after they’re deployed.
“A pass in June tells you nothing about September,” says Hulme. He explains that testing efforts need to move from roughly 80 percent of the cost of shipping an agent to closer to 99 percent to bypass any possible mishaps.
Hulme believes there should always be a human owner of an agent, and that human should be held accountable if something goes wrong. He explains: “Before you even get there, ask what the worst thing this agent could do with its access would look like, then assume it will happen. If you can’t undo an action, a human should be signing off on it, not the agent.”
According to Hulme, just because an agent scores well on a benchmark, doesn’t mean it has been verified, it’s simply auditioned. “Verification means specifying the properties you actually care about – safety, cost, fairness, staying inside its authority – and testing them under realistic pressure, not just checking it can do the task once in a sandbox. Skipping that doesn’t remove the cost of failure. It just moves the bill to incident response.”
A Rulebook for Verifying Agents
Hulme frames verification as a lifecycle rather than a one-off checkpoint – discover, triage, develop, verify, monitor, decommission – with the same test applied at every stage: does this agent have a named owner, is there evidence of how it behaves, and has that evidence been independently challenged?
For enterprises rolling out their first agents, that translates into a handful of concrete starting points:
- Give every agent an owner and a paper trail. An agent without a named owner history is unaccountable by default.
- Scope its access before you scope its intelligence. Grant the narrowest permissions and machine identity it needs, never a shared login, and require a test to justify every tool it’s allowed to call.
- Assume worst-case use of its access, not average-case. Before deployment, walk through what the agent could do if it misused every permission it holds – then build guardrails against that, not against the happy path.
- Put policy outside the model. Rules written into a prompt are a request, not a control. Controls need to sit in infrastructure the agent can’t reason its way around.
- Reverify on any change. New model, new tool, new data source – each one resets the process.
- Require a human signature on anything irreversible. If an action can’t be undone, an agent shouldn’t be the one approving it alone.
- Log and meter everything. Attribution and cost need to be visible in real time, not reconstructed after an incident.
The Takeaway
Unlike a human, an AI agent doesn’t need malicious intent to cause damage – it just needs a gap between what it was told and what was intended. Testing and verification is how enterprises can limit the amount of agentic AI incidents that occur, treating agents like the changing, unpredictable systems they are.
