Realistically, no user (technical or non-technical) really trusts an AI agent to behave itself completely as of now i.e. we’ve all seen hallucinations and biases that skew reality out of focus on the most basic “AI asks” that we perform. The UK’s National Cyber Security Centre felt AI security concerns were pressing enough to publish its own guidance on AI and agentic security this month. The threat currently circling is the jailbreak: the right mix of prompts, data, and interaction that talks a system into a bad response, or straight past the guardrails meant to stop it. What can we do to arm ourselves?
The challenge around AI security is that it is not enough to put rules and processes in place to stop jailbreaks; developers (by which we mean DevOps teams and systems architects, data scientists and all manner of AI and machine learning engineers) have to test and improve over time as new techniques or issues are discovered.
If we can do that, then we have to prove that those safety improvements actually work as intended.
Neeraj Gupta, AI/ML applied scientist at SpecterOps has much to say on this subject. For the uninitiated, SpecterOps is known for its platform that maps identity-based attack paths across Active Directory, Entra ID, and hybrid environments.
How developers really test AI
Gupta reminds us that LLM systems produce different results each time they function. To test these systems, developers have to run those prompts (or scenarios) multiple times. When a change for security gets made – for example, a guardrail gets added, or a policy is changed to prevent inappropriate responses – testing the update typically involves copied prompts, spreadsheets, screenshots, and one-off scripts.
However, advises Gupta, this approach is hard to maintain and scale up over time.
“Testing prompts also requires multiple formats and approaches to ensure that any change works in all circumstances,” explained Gupta. “Alongside a prompt in natural language, you may also look at encoding the request in base64 to see if it is recognised and a response comes back. Similarly, you can try additional prompting styles like roleplaying requests to see if responses are different. Tracking that work over time is essential, especially if you have to meet compliance checks.”
Painful parts of penetration programming
Aware that this is tough work, Gupta tells us that the “painful part of LLM jailbreak testing” is managing the whole loop around this activity.
For instance, a developer might have a few target models in place, or a combination of local models and cloud models to use, in their applications. Those systems will have different prompts in place, and have their own requirements around security or auditing. For each system, the user can apply a set of known jailbreak techniques as well as AI models that can be used to automate those attacks. To make it easier, teams can also bring in “AI as a judge model” to help evaluate responses against ‘known good’ behaviour, making it easier to scale up the number of attempts that are used in testing.
“Any testing process will log what was in place, from the target tested and what system prompt was active through to what model created the answer and the result that was returned. On top of this the system should note which tests need to be repeated and what evidence is created,” said Gupta.
Manual testing, manifold misery & malaise
Drawing upon experience gained at SpecterOps, Gupta says that, with manual operations, more time goes into listing the results rather than completing analysis and understanding the context around the responses that are coming out.
He tells us that it’s also more likely that issues will occur, skewing the results. Common issues in manual testing include missing the right prompt or re-running techniques on the wrong configuration, or not completing a test against a baseline. Similarly, getting the results together involves work to process the information and present it for others. Individual actions are easy, but scaling consistently requires a platform.
“All this describes what goes into one person’s testing process,” underlined Gupta. “But what happens when you have more than one person collaborating on testing AI? Typically, teams will have some tools and materials shared, but that can lead to problems. For example, one person might have a file of great prompts that they have developed. Their colleague may use a different set of prompts, or they may have that same set of prompts but an older version. Another colleague may have some iterative scripts that they have not shared.”
On top of this, testing techniques are all different too.
Type of testing techniques
Some techniques can be applied directly, while others might change the input; some might need an attacker model to run alongside. These tests might need to be run multiple times to gather enough data to be useful. When it comes to testing LLMs and jailbreaks, many test results have to be presented in context in order to be useful.
Gupta provides deeper insight here and says that “looking at the test result on its own” is less useful than seeing the original intent, techniques and prompts alongside the model configuration that was in place at the time. That kind of AI Bill of Materials is needed if developers want to reproduce the same circumstances in the future, or to point to why things were changed.
“AI security testing is developing rapidly alongside AI development. As more AI systems get deployed, we need the same rigour around keeping these environments secure that more traditional applications have,” said Gupta
For him, the “interesting work” around jailbreaks is not getting one model to answer one bad prompt. It is being able to run the same checks again, compare results, review evidence, and understand whether your changes actually improved the system.
“Standardising AI and LLM testing is a necessary step to prove that these systems are managed and kept secure, and this makes it easier to automate and scale the process up. It also provides these results for compliance, as you can point to a process that provides before and after mitigation data against baseline performance,” concluded SpecterOps’s Gupta.
Neeraj Gupta works on AI/ML applications in security research, he developed Jailbreaker as an open-source tool for managing LLM security testing and research.