It was the summer of the AI agent going haywire. An astounding number of organizations were affected by agents running on AI models still in training or testing at the likes of OpenAI and Anthropic, with OpenAI alone notifying more than 100 organizations. At GTC Berlin, Nvidia CEO Jensen Huang highlighted how his company is putting the AI genie back in the bottle.
The main thrust of this initiative is the Open Agent Safety Platform, launched on September 28. It consists of the OpenShell software runtime as well as Sentry, a watchdog at the hardware level. Asked by Techzine, Huang explained that OpenShell specifically will “precisely prevent the incidents that I think you and I are talking about.” This ambitious goal revolves around a more fine-grained control layer than current methods, which realistically were only suitable for classical applications.
Breaking containment
A point often made about the Hugging Face hack by OpenAI agents and the multitude of other examples is that “malice”, if AI can possess such a thing, was largely absent. Agents were given tasks such as research and cybersecurity benchmarks and found ways around their containment, in OpenAI’s case with safeguards deliberately lowered for testing. Huang explained that today’s containment is essentially a virtual machine with static rights. An agent gets access to a file, a directory, a network, or simply everything. Instead, “you need to have extremely granular policies for [an] agent,” Huang said.
Critically, one needs verifiable proof that containment works as intended. “It’s called formal verification,” the Nvidia CEO explains. This revolves around mathematically proving that a system meets its specification, instead of relying on sample cases. In OpenShell, a “policy prover” checks the rules an agent proposes and flags risky new access for human review. A boundary check does as the name suggests: it proves that a policy never grants an agent more reach than a maximum configured beforehand.
Going deeper
Nvidia’s approach is derived from its rather expansive hardware selection. Beyond its GPUs, the reference design draws in Nvidia’s Vera CPUs, on which OpenShell is tuned to run, and its BlueField-4 DPUs, which host Sentry. Sentry runs separately from the machine executing the agent and can quarantine an agent within milliseconds if it crosses its boundary. This hardware layer is optional: OpenShell can be adopted on its own and also runs on Arm and Intel chips. One may wonder why this company specifically is so intent on AI safety. After all, isn’t it up to the AI labs to check their own work?
Anthropic has already integrated OpenShell into its Claude Managed Agents, and Huang lauds the AI labs’ efforts to do better. He explains that “safety is the number one discussion, number one conversation in AI today, and largely it has to do with making sure that we have a secure containment system for the agent.”
While OpenShell might have prevented or mitigated the AI agent breaches, Huang rightly points out that there was a more fundamental issue: some agents simply weren’t being monitored. Reuters reported that OpenAI did not notice its agent hacking Hugging Face for a week, while an agent’s breach of an Australian Medicare portal on June 18 went unnoticed by OpenAI until August 11.
Bullish against slowdown
One question from Time triggered a rather feisty response from Huang. Several AI lab leaders, including Anthropic CEO Dario Amodei, have called for ways to deliberately “pace the frontier” of AI development. A legally enforced and coordinated slowdown is emphatically not on the Nvidia chief’s mind. “That’s ridiculous,” he noted. “I just want to make sure we are adults here.” Industry-wide norms, standards and audits are being developed, Huang stated, but if an AI lab, any AI lab, already knows that its model delivers harm, it should “just stop”.
Huang instead suggests policymakers should focus on the applications that use AI, rather than the technology as a whole. With Europe leading the charge on regulating the technology, Huang warns that stifling the industry altogether would leave Europe with “no AI at all”, which he said “could be the worst outcome.”
Also read: Nvidia GTC Berlin: on the hunt for European AI factories