Microsoft CEO Satya Nadella (photo) advocates a fundamentally different approach to enterprise AI security. As more and more autonomous models gain access to sensitive data and critical systems, organizations must treat those models as potential internal threats rather than as trustworthy technology.
In a post on X, Nadella states that traditional approaches to software security are no longer sufficient. Traditional software allowed developers to trace specific behaviors back to specific code paths. With advanced AI models, this level of transparency is lacking. Their output cannot be reliably attributed to specific training data or to the model’s configuration.
Despite this uncertainty, organizations are now rolling out AI agents capable of accessing confidential information and taking action without human oversight. Guarantees from AI providers cannot replace organizations’ own responsibility when it comes to controlling these systems.
Separating intelligence from authority
Nadella proposes separating the intelligence of AI models from the authority to act. Access rights, operational boundaries, and security measures must be independent of the AI models themselves.
This implies that the AI model must be separated from the software that controls the activities and actions of agents. This approach builds on established principles within cybersecurity, including the “least privilege” principle, identity management, and activity logging. Nadella argues that this practice should be extended to AI systems, regardless of whether they are proprietary or open-source models.
Nadella also questions whether AI can reliably monitor itself. Using one AI model to monitor another can enhance security, but it can also lead to opaque decision-making. Even insight into a model’s thought process is not sufficient, because the explanation does not necessarily reflect the internal decision-making process.
External controls and emergency shutdowns
Nadella outlines several requirements for the safe deployment of AI. Organizations must avoid relying on a single model for critical decisions. They must maintain logs of the models’ actions and continuously test systems for attacks and unexpected behavior.
Independent oversight is of the utmost importance. Models must not be allowed to manage both their own actions and the evidence used to verify them. Companies must also take into account that models can be compromised. Authorized personnel must be able to interrupt or terminate an agent’s execution of a task.
Nadella’s key conclusion is that the most reliable AI system will not necessarily be the most reliable model, but rather the system that urges us to trust it as little as possible.