Fox-IT has detected a major exploitation campaign on Citrix NetScaler servers. In cooperation with the Dutch Institute of Vulnerability Disclosure (DIVD), the company is notifying victims. The attackers were able to compromise a huge number of servers in merely two days with the help of automation tools. Many backdoors still exist, including at organizations that have already patched the vulnerability and falsely believe themselves to be safe.
Vulnerability CVE-2023-3519 came to light in July. It allowed Citrix NetScaler servers to be deployed to perform remote code execution in an enterprise environment. Typically, companies use these servers to facilitate HTPP redirects and rewrites, set up a firewall or provide load balancing, for example.
The vulnerability was actively exploited, including by a still-unknown party that carried out a campaign with a global impact on July 20 and 21. Using automation, the malicious actors installed Web shells in order to later return to corporate networks. Citrix soon came out with a patch and was able to fix the problem for organizations that had not yet been affected. However, it is quite possible that an IT environment already houses a backdoor: the attackers moved quickly, so patching was often too late. It’s not easy to tell if a company’s been compromised. Thankfully, Fox-IT and Mandiant have a solution for this.
Rescuer in distress
To help organizations figure out their status, Fox-IT rapidly produced a Python script that uses its own Dissect tool to provide triage on forensic images from NetScalers. In other words, it can examine a 1-to-1 copy for signs of backdoors. The reason Fox-IT was able to respond quickly was because it already had expertise in Citrix vulnerabilities and the company’s server architecture. Mandiant also provided a bash script that looks for backdoors. Both variants can be found in the Fox-IT blog.
The problem is not simply fixed in doing so: because 69 percent of all compromised servers have already been patched, many organizations will not realize they are still susceptible. DIVD has already been busy informing victims, but it’s hard to imagine they’ll be able to reach everyone.
The attack campaign: impact bigger than expected
When Fox-IT went looking for the impact of the attack campaign, they were surprised by its size. 31,127 NetScalers were vulnerable on July 20-21. Nearly 2,000 unique IP addresses were backdoored through a Web shell. Of all the vulnerable NetScaler servers, 6.3 percent were actually affected. These instances occurred worldwide, with a so far unexplained center of gravity in Europe. The attackers were most active in Germany, France and Switzerland. In the Netherlands, there are reportedly more than 100 known NetScalers with a backdoor, 82 percent of which have been patched. Again, that doesn’t mean the coast is clear.
1828 NetScalers still contain a backdoor, of which 1248 have been patched.
The lesson Fox-IT draws is this: any edge device should receive patches as soon as possible, because the chances of something going wrong quickly are high. As with a major cyber threat such as Log4Shell, attackers move quickly with automated scanners. They then do not even have to lift a finger in serious cases to strike thousands of organizations. Those who do not know they are at risk can then be exposed at any time thereafter.