Log4Shell hit the market hard at the end of 2021. According to a recent story we published, it’s still one of the biggest vulnerabilities, 18 months after it initially appeared. How serious is the threat of Log4Shell in 2023?
Log4Shell continues to haunt us. The exploitation of a vulnerability in logging software Log4j caused quite a stir in late 2021. Almost every organization was at risk, whether they were aware of it or not. In order to find out how serious the vulnerability still is, we asked the security industry to provide us with their insights. How big is the risk that organizations still run? How many exploits are there still in the wild? And above all, what can and should organizations do about Log4Shell in 2023? And provide us with input the security suppliers in the market did. We received a lot of it.
We hope you like the story we constructed out of all of the input we received. It’s a first of many more of this type to come. So if you have suggestions on topics, please let us know!
Big impact in late 2021
First, it is important to highlight how widespread the Log4Shell threat was. Up to 93 percent of all cloud environments were susceptible to the exploit. “Log4Shell is one of the biggest cyber vulnerabilities ever,” Steve Stone, head of Zero Labs at Rubrik, puts it bluntly. It also kept the people over at Cisco busy over the year and a half. Jan Heijdra, Security Specialist at Cisco, sheds some light on this. “Despite the fact that the vulnerability has been fixed for over 15 months, it turned out that 72 percent of organizations saw a Log4Shell alert on their firewall in 2022. With an average of about 110 Log4Shell alerts per month at an average organization, Log4Shell dominated the rankings in 2022.”
Also read: Log4Shell: what is Log4j, who does it affect and how do you patch it?
A large number of applications were for the Log4Shell exploit. The Dutch NCSC (National Cyber Security Center) maintains a list of affected applications on GitHub. On the one hand, it is startling to see how extensive the list is. On the other hand, many vendors have patched their applications by now. “You have applications that were known to be vulnerable to Log4Shell. If you use these applications and make them publicly available over the Internet then there is a very good chance that your system has already been compromised in the months after the leak became public,” states Daan Keuper, Security Specialist & Head of Research at Computest. “This is because criminals use automatic scanners to scour the entire Internet. So companies affected by this have probably already felt the effects.”
Log4Shell evolves
The use of scanners works both ways. Not only the presence of the Log4Shell vulnerability itself, but also incidents of its misuse come on experts’ radar thanks to detection systems. What this shows is that there has been a change in behavior since the vulnerability became known in late 2021.
EMEA CTO Zeki Turedi of CrowdStrike has seen this happen. “Log4Shell, in part due to its notorious and long-standing nature, was the most prominent example of vulnerabilities discovered in 2022. Log4Shell was initially opportunistic in nature: cybercriminals searched for vulnerabilities and focused on what they could find. But variations of the existing exploit soon appeared, targeting other areas and using other protocols and techniques to remain invisible.” This, according to Turedi, made it possible for attackers to use the vulnerability on products where it was not initially possible. This is what cybercriminals were in constant contact about. “Falcon Intelligence Recon observed ongoing CVE-2021-44228 discussions between cybercriminals in 2022,” Turedi points out. “This indicates continued interest in exploiting the Log4Shell vulnerabilities.”
This specialization has meant that vulnerabilities may still be current today, we hear from Fox-IT. “In the first months of Log4Shell, the vulnerability was massively exploited. During that period, an attacker may have inserted a backdoor. Such a backdoor can often continue to work even after a system has been patched. This allows an attacker to return at a later time, even if a system is no longer vulnerable to Log4Shell. We have encountered this type of backdoor several times in incident response, and it is likely that we will encounter it again in the future.”
Log4Shell in action
We received some interesting examples of Log4Shell from Alex Hinchliffe, Threat Intelligence Analyst at Unit 42 by Palo Alto Networks. “The espionage-motivated threat actor Boggy Serpens (also known as MuddyWater, MERCURY/Mango Sandstorm) was attributed by CISA (Cybersecurity & Infrastructure Security Agency) to the MIOS (Ministry of Intelligence and Security) of the State of the Iranian state. This was detected by Microsoft when they exploited this vulnerability when attacking organizations in Israel in July 2022.”
At Unit 42, in addition to the above threat, they also saw an attack on a telecom company in Israel via a Log4j vulnerability in SysAid software. This took place in January, March and again in May 2022, Hinchliffe indicates. Unit 42 saw the same group exploit the same vulnerability when it tried to infiltrate a telecom organization in Kuwait.
Annoying but not always disastrous
In many cases, malicious actors have primarily annoying effects on an organization, but not necessarily disastrous ones. Director of Product Marketing Amit Shah of Dynatrace said, “Some companies are still vulnerable to Log4Shell because they have not yet been able to find and patch all instances of Log4j. For example, a new attack campaign involving ‘proxyjacking’ was recently discovered. This involves attackers trying to install proxyware, then selling Internet bandwidth from victims. While this is a nuisance rather than a serious threat, it can still cost victims dearly when a cloud provider charges fees based on measured data traffic. In addition, a victim’s internet bandwidth could be used for a cyber attack.”
The proxyjacking example Shah gives shows something important, according to him: “It shows that attackers only need to find one unpatched instance of Log4j in a container to take it over.” The fact that many organizations use a lot of commercial software also makes it difficult to detect. These often do not have access to the source code of commercial software, Shah points out. This makes it harder to scan for vulnerabilities. “Only fixing vulnerabilities in proprietary applications that use Log4j is therefore not enough,” he concludes.
Also read: New cybercrime tactic: trading other people’s internet connection
Simple exploit, not always interesting for attackers
From the above, you might conclude that the danger of the Log4Shell vulnerability is not that great. This is true to some extent. Nevertheless, it is important to add nuance to this as well.
From Sygnia, MD Northern Europe & EMEA Azeem Aleem provides this nuance, when we ask him about his view of the current state of affairs surrounding Log4Shell. “We see that Log4j is still widely exploited today, due to the vast array of applications which utilise vulnerable versions of the Java based logging utility.” In other words, companies still have to patch those. Aleem sees that there is often a link between Log4Shell and botnets, “with an end goal of installing cryptominers such as XMRig.” However, he also notes more serious “use cases” there. Indeed, he certainly sees sophisticated malicious actors still using Log4Shell “to gain initial access to specific target organisations of interest.”
John Dwyer, Global Head of Research at IBM Security X-Force also has an interesting insight on this point. “In the majority of the Log4J cases that X-Force has responded to, no malicious activity was detected. Organizations were able to discover the vulnerability early on through automated scanning.” He further notes that X-Force’s investigation revealed “that in most incidents, the vulnerability had
been exploited but there was no follow on activity by threat actors.”
Log4Shell is No. 1 in the top 10 of exploited vulnerabilities that we know about, according to senior SOC analyst Edwin Tump of Pinewood. At more than 20 percent, Log4j is the lone front-runner. “It’s important to note that these are attempts; it’s not about successful exploitation of vulnerabilities.” Pinewood sees little to none of those.
The data: what are we seeing now?
We have found so far that Log4Shell is still far from dead anno 2023. We have also found that the severity of the vulnerability is relatively not that bad. Any vulnerability that attackers can exploit is obviously one too many. However, there are considerably more serious vulnerabilities than Log4Shell, even though it is thus still very dominant.
To dive a little deeper into Log4Shell in addition to the general observations above, it is interesting to take a look at some data. This data was provided by Fortinet, with Gergely Révay, EMEA FortiGuard Labs Systems Engineer also providing the necessary interpretation of this data.
For Log4Shell, Fortinet has implemented several protections in its products. Of these, the Intrusion Prevention System (IPS) in FortiGate is the most interesting, Révay points out. Remarkably, a specific ‘variant’ is dominant. Of three common “signatures” to see Log4Shell-related attacks in IPS, Apache.Log4j.Error.Log.Remote.Code.Execution is by far the most frequent. “On average, we can say that in 2022 this signature was the most triggered IPS signature globally from all signatures. For instance, Figure 1 shows the top 10 detections in the Netherlands for Q1 2022.”
We see that Apache.Log4j.Error.Log.Remote.Code.Execution tops the list. “This was the same for most countries,” indicates Révay. Figure 2 shows how the detection numbers change during 2022: “Instead of slowing down, we can see that detections increase at the end of the year. Altogether we had 88 billion detections for the three IPS signatures globally.”
By 2023, Log4Shell does appear to be on the decline, we infer from the data Fortinet shares with us. You can clearly see that below. Small note here is that we got the input from Fortinet toward the end of April. So that month is not yet complete. However, the trend seems clear. After January, the number of detections collapses considerably.
Révay takes an additional look at the Netherlands. Figure 4 shows the top 10 IPS signatures for the Netherlands in Q4 2022 and Figure 5 for Q1 2023. In both cases, the Log4j signature is in the top three.
The fact that the number of detections of Log4Shell is declining does not mean it is on its way out, Révay concludes: “Although we see ups and downs in our detection telemetry for Log4Shell, it is one of the most detected attacks globally. We don’t expect this to change in the near future.”
In the end, attackers often take the path of least resistance anyway, we know from past experience. Log4Shell clearly still is. Révay gives a possible reason for this. This may be “that it is a vulnerability that could be challenging to track down because software might not use Log4j directly.” Log4j may then “still be used by a dependency.” This can be an adjacent dependency, but also one further down the dependency chain, according to Révay. He therefore has some clear advice for software vendors. They “should track down their dependencies to ensure they are not vulnerable.”
We’ve had the worst of it…
Based on Fortinet’s numbers, it doesn’t seem like a crazy observation to us that we’ve had the worst of it. However, opinions differ somewhat on that. Fox-IT’s teams seem to agree: “Both Fox-IT’s Security Operations Center and Incident Response team see a decrease in exploitation of Log4Shell. While attackers still do a lot of scanning for the vulnerability, we see that most systems have now been patched. Whereas in 2022 there were still multiple ransomware attacks linked to Log4Shell, in 2023 there are hardly any more.”
Daniel Thanos, Head of Arctic Wolf Labs, agrees. “In recent months, we have seen a decline in Log4Shell activity. Based on our telemetry, the number of unique incidents of scan and exploit attempts has decreased since the beginning of 2023. We observed 7,386 unique incidents (totaling about 10 percent of our customer base) in the last four months of this year, compared to 63,313 incidents (totaling 25 percent of our customer base) in the 10 months we talk about in our December 2022 blog.” So that covers most of the first year of the Log4Shell vulnerability.
Steven van Gysel, Manager Solutions Architect Northern Europe at Infoblox, draws a similar picture. Among Infoblox’s customers (mainly the larger organizations), they see that “Log4Shell is no longer a hot topic there as it was in December 2021. Then it was really all hands on deck, but now we see activity mainly from red teams or threat researchers.”
By the way, Van Geysel does see another use of Log4Shell by these professionals. “They are still using this to test their security stack and older malware. But they also use the threat intelligence to fight newer malware, as we saw recently with ‘Decoy Dog,'” he gives as an example.
Finally, Check Point Software Technologies has the clearest statement about the threat of Log4Shell. Through Zahier Madhar, that company lets us know that the current threat landscape does not show any large-scale threats focused on Log4j.
…or did we?
Stefan van der Wal, CSA Application Security at Barracuda Networks doesn’t quite agree with the sentiment that Log4Shell is on its way out. In any case, he doesn’t want to cheer too loudly just yet. “Log4Shell is now almost 1.5 years old, and we may think the worst is behind us, but in fact it is still a huge problem. Our own scanning systems have detected as many as 5 million patterns indicating attempts to reach Log4Shell in the past 30 days alone [gauge date in late April, ed.]. This means that attackers are still very actively looking for vulnerable systems and they will only do so if there are still many Log4j instances that have not yet been patched.”
Cisco’s Jan Heijdra underscores this point. “As the data show that attackers continue to use vulnerabilities that are several years old, it is likely that Log4Shell will remain an ongoing problem for a long time to come.”
Also read: Log4Shell still a major problem after nearly a year and a half
Thus, conclusions about Log4Shell activity vary, but indicate a significant reduction anno 2023. Still, criminals continue to scan for the vulnerability. Even though organizations should have fixed them by now. However, the overwhelming popularity of Log4j in software will keep Log4Shell relevant for an extremely long time to come.
How do you deal with it as an organization?
Now that we have a relatively good understanding of the current state of affairs surrounding Log4Shell, let’s also look at how organizations deal with it. Or at least how they should deal with it.
The problem with the Log4Shell vulnerability is that Log4j can reside just about anywhere. There is simply a lot of software that uses it. Organizations have very poor visibility into this. The vulnerability in itself may not score off the charts (anymore) in terms of severity, but the fact that it is so hard to find partly compensates for that. It is very difficult for companies to know quickly what risks they face, even in 2023.
Aleem from Sygnia talks about “analysis paralysis syndrome” in this regard. Organizations do not know exactly where the problem is and which applications are vulnerable. “So they remain in a state of limbo and don’t know where to begin,” he states. This, he says, is the main problem surrounding Log4Shell for organizations.
It’s 2023 and we’re still talking about…patching
The solution that Aleem and others propose is as old as the road to Kralingen. “It all comes down to patch management,” he points out. “First, you have to critically review all applications to determine which are affected and then remediate. It’s critical for organizations to track what is running in their environments and have an ownership and inventory plan and process for all applications.”
Deepen Desai, Zscaler’s CISO, points out the biggest problem: “The Log4j vulnerabilities have caused a focus on vendor management and supply chain security.” The main issue here is the applications that organizations use. Today, that is a mix of self-developed and open source components, he points out. We’re not doing enough to get a clear picture of what people are using, he believes. That is a “systematic security problem,” according to him. It results from “the continued failure of organizations to identify their assets,” but also from “ineffective vulnerability management and patching and supply chain security.”
Desai also makes a prediction at this point if organizations do not improve this. “This decentralized and diverse software supply chain, which relies heavily on open source projects and code, ensures that vulnerabilities such as Log4Shell will become much more common unless they are managed effectively”, he says.
Barracuda’s Van der Wal also speaks of an ongoing patch problem: “Many organizations have chosen to patch only public systems for now. This means that there are still a lot of vulnerable systems that may not be directly accessible from the Internet, but where attackers who have managed to get onto the corporate network via another route can therefore still make their move, with all the consequences that entails.” He emphasizes again that this will lead to more problems. “So we can expect more Log4Shell attacks and that’s something companies should be wary of,” he warns.
To indicate how poorly patching is occurring, Van der Wal also cites some figures. “Another concern is that software supply chain management specialist Sonatype noted in late 2022 that a year after the fix for Log4shell became available, as much as 30-40 percent of all Log4j downloads still involve the vulnerable version,” he points out. He also points to the continued availability of the older, vulnerable versions of Log4j. As long as they’re available, they might end up in production environments. It might help if those were no longer available.
Gaining insight into risks
In itself, it is clear what companies need to do (and in fact should have already done) when it comes to Log4Shell. Pieter Molen, Technical Director at Trend Micro Benelux, sums it up. “To understand the risks that these vulnerabilities can cause and to address them, first of all you need to know what elements are being used across the IT ecosystem.” These include workstations, servers, network and IoT components, cloud environments, SaaS applications and, of course, application modules such as Log4Shell. “All of these elements make up the digital attack surface. Understanding the digital attack surface and potential security risks are the basis for taking appropriate measures to prevent vulnerability issues.”
However, it’s not easy to get this insight. As such, not all organizations can or will take the required steps. Molen more or less observes this himself. “Unfortunately, we see that the Log4Shell vulnerability is still being abused to gain access to organizations. Extra painful here is that organizations are often not even aware that this application module is being used within their organization until it is too late.”
Log4Shell is ‘endemic’
Clearly, many organizations have their work cut out for them. Detecting application vulnerabilities may sound easier than it is. On the one hand, the relationship between proprietary and open-source elements is often unclear. That already creates confusion. In addition, by no means every party reports that their software is vulnerable to the Log4Shell exploit. This may be because it no longer provides support for it. However, it could also be that a software vendor does not even know that a component related to Log4j is present somewhere. Finally, a software vendor may also be bankrupt, of course.
For that reason, Log4Shell is here to stay for the foreseeable future. Zscaler’s Desai cites the judgment of the Department of Homeland Security. This part of the U.S. legislature calls Log4j an “endemic vulnerability”. It expects that vulnerable copies of Log4j will be part of systems for many years to come.
A change is gonna come
The use of terms familiar from the corona pandemic are not very reassuring. Yet the corona pandemic also came to an end. The same will eventually happen with Log4Shell. There is light at the end of the tunnel already. At least that is what we gather from the words of Stone of Rubrik Zero Labs. “The discovery of Log4Shell functioned as a wake-up call,” he indicates. He also says there are now reasons for optimism about the future of the cybersecurity landscape.
Stone explains what he means by “reasons for optimism”. One is that the cybersecurity industry now realizes there is a problem. “Not so long ago, security breaches were a taboo subject,” according to him. “However, the number of cyber incidents in recent years has caused cybersecurity to become a board-level topic,” he observes. “In addition, cybersecurity education has also improved. More and more major universities now offer cybersecurity master’s programs where students learn about data breaches and vulnerabilities that are happening now – today’s cyber attacks are tomorrow’s case studies.”
Finally, specifically for the Netherlands, Stone sees positive developments. According to him, the Netherlands is “paying increasing attention to cybersecurity legislation. In addition to the upcoming NIS2 directive to increase cybersecurity at the European level, the Dutch government is also engaged in several studies to explore national information security legislation.”
Textbook example
All in all, we will learn a lot from Log4Shell, and we should. It will undoubtedly continue to serve as a textbook example for software vulnerabilities for decades to come. For now, it is important that companies roll up their sleeves and continue to check that their software suite is up-to-date. Those who are vigilant when it comes to the software used within their IT environment can minimize the chances of a Log4Shell incident. And that’s what it’s all about after all, risk management. With that, the problem of Log4Shell anno 2023, while not completely solved, is manageable.