GenAI could play a major role in the field of security. Google researchers base this finding on an experiment with Gemini 1.5 Pro. Still, there are pitfalls.
The generative side of AI systems like Gemini and GPT-4 sometimes overshadow their analytical capabilities. For example, where ChatGPT continuously creates insecure code, LLMs prove to be just right for analyzing existing programming code.
Experiment
At least, that is the general conclusion of a Google study shared by the company today. The test model chosen is Google’s own Gemini 1.5 Pro. This LLM particularly excels when it comes to its context window, essentially the short-term memory of an AI model. Gemini 1.5 Pro can take in up to 2 million tokens. That means it scans large codebases or processes multiple files simultaneously with ease. As a result, Gemini perceives “a deeper understanding of complex relationships and patterns within the code” than models with smaller context windows. At the time of writing, that’s every other LLM; none of the alternatives come close to Gemini’s 2 million.
The potential of AI scanning for vulnerabilities is great. Thus, the researchers see an opportunity to take vulnerability detection “beyond surface-level flaws.”
Methodology
For the study, the Google team built what it calls a software vulnerability scanning and remediation engine. But what does this entail? In essence, it’s an interplay between Gemini 1.5 Pro on Google Vertex AI via the Python SDK, data from a Google Cloud Storage bucket and a compression of the information from the bucket followed by prompt engineering. Google isn’t sharing the exact prompt, but it says the end result is a report in the form of a JSON or CSV file.
Still, the current implementation is not yet mature. The Google team warns others that data is not anonymized through this method and advises consulting legal and security experts before an actual implementation of AI vulnerability scanning can take place. More research is needed to arrive at a production-ready detection engine.
Also read: Smartphones vulnerable due to Qualcomm GPU drivers, Google discovers