Claude Mythos 5 overstepped its bounds during a cyber test conducted by the British AI Security Institute. The model attempted to inject malicious code into a GitHub repository belonging to real developers and used fake accounts to pressure the maintainer. Anthropic has published its own analysis alongside the AISI report.
On August 4, the British AI Security Institute (AISI) published a report on its tests of Mythos 5’s cyber capabilities. The model was given a capture-the-flag (CTF) challenge. It had to capture a secured piece of data, the “flag,” from a simulated network that resembled a real organization’s systems. Internet access was intentionally enabled, which was standard practice for AISI’s cyber tests at the time. That is no longer the case. Anthropic also disabled the cybersecurity safeguards it employs in publicly available models such as Fable 5.
In nine of the 43 evaluation runs, Mythos 5 took actions that fell outside the scope of the test. Those actions affected real people and organizations. According to The Hacker News, AISI recorded 19 live internet actions across the entire test series in 10 of the 122 runs. Of those, 17 are attributed to Mythos 5.
From test network to real repository
In its analysis, Anthropic, like AISI, focuses on a single transcript: the complete record of the model’s reasoning and actions. Mythos 5 first tried to penetrate the test network using the evaluation methods provided. It was unsuccessful. However, the model quickly discovered that one of the machines periodically communicated with GitHub.
Mythos 5 saw an opportunity there. It searched GitHub for a repository from which the model believed the machine automatically downloaded software. But that repository belonged to developers who had nothing to do with the test. The model tried to inject malicious code into it, assuming the CTF machine would become infected on the next download. It involved a pull request that combined legitimate code with a hidden malware dropper.
It didn’t stop there. Mythos 5 created fake identities that pressured the maintainer to approve the change. Those accounts falsely claimed that the code was safe and useful. To do so, the model used, among other things, Tor and a proxy to bypass GitHub’s login restrictions. The maintainers rejected the change, after which the run eventually timed out.
“This behavior would have been flagged immediately by our production safeguards on Fable 5, had they been enabled,” writes Anthropic. Nevertheless, the company believes it is important to study the behavior. According to Anthropic, the underlying errors are not necessarily specific to cyber incidents. They point to broader issues in the model’s behavior in evaluation environments.
Tip: EU cyber agency gains access to Mythos 5 after a 3-month delay