GitLab’s AI assistant Duo may have been complicit in code theft. The bug in the code for rendering HTML has been fixed, but the risk remains.
The dangers extend beyond code theft. Dark Reading notes that the spread of malware and so-called “dirty links” to malicious sites was possible. Legit Security discovered an indirect prompt injection that made these dangers possible.
The cause is clear: Duo apparently does not scrutinize the input it receives critically enough. It appears to be a fundamental security flaw that actually has little to do with the AI nature of Duo.
Duo as a danger
Duo is a tool within GitLab that users of the open-source repository can use to analyze code, suggest changes, and automate certain aspects of writing, testing, and merging code. In other words, it is an alternative to GitHub Copilot, as Dark Reading also noted.
This vulnerability is particularly worrying because GitLab has already had to deal with serious security issues in the past. At the beginning of last year, for example, thousands of GitLab instances were vulnerable to account takeover due to a previous critical vulnerability that allowed hackers to send password reset emails to email addresses they controlled.
Broad attack surface
Because Duo affects every aspect of GitLab, manipulating it is extremely effective. The tool does not discriminate between source code, commits, descriptions, and comments. Malicious hidden prompts can be embedded in any component.
Duo executed hidden prompts that were injected everywhere users interacted with the site. An attacker did not even need to infect their own code, but could attack others via comments. Attackers were able to hide the prompts by, for example, combining white text with a white background.
HTML injection via Duo
The biggest danger came from the way Duo generates its responses. Duo formats its responses in Markdown and displays them line by line, generating HTML for a browser to read as a stream. However, it renders the HTML as it goes, without waiting for the entire response to be rolled out.
GitLab has since released a fix for this HTML rendering vulnerability. Earlier this year, the company launched GitLab 18, which brought Duo Enterprise to Premium customers, with improved AI functionalities. Whether this update also addresses the security issues has not been disclosed.
Prompt injection remains a risk
However, the company has not addressed the other prompt injection risks that allowed researchers to poison Duo’s responses. GitLab told Legit Security that it did not consider this a security issue because it does not directly result in unauthorized access or code execution. Security researchers disagree. Anyone affected by the vulnerability will quickly come to the same conclusion.