Techzine had an exclusive interview with Fermín Serna, the Chief Information Security Officer (CISO) of Citrix, with the most important question; where did it go wrong? Citrix has dominated the news for the past week with a critical security vulnerability in Citrix ADC and Citrix Gateway. The problem seemed to become bigger every day. We asked the CISO several questions about how this situation escalated and what lessons can be learned from it.
Before the interview with Citrix, we critically analysed the security leak, looked at the initial reports, media attention and the mitigations offered by Citrix. We also tried to determine when things really went wrong. Our first conclusion is that this story is anything but unique; what happened to Citrix can happen to any company. There is a domino effect here, where the situation is getting worse and worse over time. The low point is probably that Dutch radio reported about Citrix traffic jams because people had to work in the office instead of at home. In the end, it’s about the lessons which can be learned from this.
What happened?
For those who didn’t follow the Citrix news, we went through the process and decision making process step by step with the CISO of Citrix, Fermín Serna. To find out what went wrong, and whether it could have been prevented.
The beginning of the story around this very critical security vulnerability already starts in mid-December. Until now, it seemed that Positive Technologies was the only company that had discovered and reported the security leak to Citrix. During our conversation with Serna, he corrected us. Positive Technologies reported the security vulnerability, but there were two other security researchers from other companies who made the same report in a time frame of two days.
It is not clear how three different security researchers can make the same report in a period of two days, especially when it concerns a security leak that has likely been in these products for quite a while. However, we do know that security researchers often work together and share knowledge. Perhaps they shared insights?
What did Citrix do with the three reports?
When a security breach is reported to a major vendor like Citrix, the first step is to verify the vulnerability and then perform an analysis. That’s where they look:
- How big (critical) is the vulnerability;
- Is the leak already being abused by malicious parties?
- How big is the chance that it will become public knowledge?
- Are there possibilities to keep it quiet for a couple of weeks or months, so the vendor can develop a patch?
This is the usual procedure for almost every large software company. There is also a kind of industry standard that offers 90 days to develop a patch, during which the leak will not be made public.
Citrix also works according to this procedure, but in this case, there was not one party, but three parties that reported the same security leak. That made the situation a lot more complex. The chance that the security vulnerability would be made public, a so-called uncontrolled publication, is many times greater with three parties.
According to Serna one of the three security companies informed Citrix they were not going to give Citrix a waiting period at all; the security firm stated that they would publish the vulnerability on December 23rd. Serna didn’t mention any researcher or company names. We have noted though that Positive Technologies have published a very extensive blog on this exact date. So, for Citrix, there was no way to develop a patch and keep the security leak quiet for a few more weeks.
We also asked if any of the security companies asked money for a waiting period so that Citrix could develop a patch. We received a firm “no”.
Citrix published the leak and a temporary solution to close it
Citrix’s options were limited after it knew that a publication would follow shortly. In such a short period, it is impossible to develop a solid patch, test it well with all the different versions and distribute it to customers.
Citrix decided to take the lead and published the leak itself on December 17. With this publication, Citrix also supplied temporary mitigation to close the security leak. This mitigation consists of several commands that have to be executed by administrators on the Citrix servers. These commands consisted of two blocks, that both should be executed.
This temporary mitigation ensures that it is no longer possible to abuse the security breach, but keeps the Citrix servers up and running so that users can still log in (also externally). The temporary solution, therefore, has no negative effect on the use of the Citrix servers.
“The temporary mitigation works on all Citrix versions if both blocks of commands have been executed.”
We asked Serna about the media reports that this temporary mitigation wasn’t working on all Citrix servers, that the security leak would not work on certain versions or builds. Serna indicated that those reports did not escape Citrix, and they did additional research. They concluded that the mitigation works on all versions and builds when both blocks of commands are executed. Not just the first block. So it seems that some Citrix administrators did not execute both blocks of commands.
Analysis of the security vulnerability and temporary mitigation
We analysed the security vulnerability as well as the temporary mitigation. The security vulnerability can be abused by manipulating a URL to execute commands on a Citrix server. If we look at the temporary mitigation, we see that if there is a URL request towards “/vpns/”, which is a default directory on Citrix ADC and Citrix Gateway, combined with “../”, then the server should send a 403 forbidden (kill the request). There were a few more commands, but this is the most important one.
Based on this rule, Citrix already more or less shows in which direction to look for this vulnerability. We asked Serna if this could not have done differently? Now it looks a bit like Citrix reported on December 17 that the door was open with a kind of treasure map pointing in the right direction. According to Serna, Citrix carefully weighed its options before publication and opted for a balanced solution, a solution where the customer is fully protected and where as little information as possible is shared, not to tip off people with malicious intentions. That is why the forbidden rule is as simple and as broad as possible.
On January 8 an exploit got published to exploit the vulnerability
The temporary mitigation from Citrix worked well for more than two weeks, but after that, it quickly went downhill. It was clear that there was a major security vulnerability in Citrix; this attracted the attention of malicious people and security researchers looking for attention. Eventually, more and more reports and blogs were published by security researchers that figure out how to exploit the vulnerability. They showed screenshots and proof of the vulnerability, in most cases, they used black bars in their images, but according to Serna, these screenshots all gave away hints on how to find the vulnerability.
On January 8, the vulnerability becomes even bigger when an exploit is published, which shows exactly how the security leak can be abused. What follows are even more security reports about how there is a hunt for vulnerable Citrix servers. Analyses with numbers of vulnerable Citrix-servers per country. Government agencies feel urged to come with warnings. That creates an additional wave of media attention where the Dutch government eventually decides to shutdown the Citrix servers. The lowest point is reached when Citrix traffic jams are reported on Dutch radio because people have to work in the office instead of at home.
“If Citrix administrators had implemented the temporary mitigation in the days following our publication on December 17, they would have been protected against this security vulnerability.”
It is a domino effect that cannot be stopped and where people fall over each other to blame Citrix. What people forget is that in the end, security is a shared responsibility.
Or as Serna puts it, somewhat simpler: “If Citrix administrators had implemented the temporary mitigations in the days following our publication on December 17, they would have been protected against this security vulnerability”.
Patches available
The first patches for this vulnerability appeared on Sunday, January 19. However, the solution is only available for versions 11.1 and 12.0. The patches for the other versions will appear on Friday, January 24.
What we didn’t quite understand is that Citrix has chosen to patch these versions first. There is also version 10.5, 12.1 and 13.0. Where 13.0 is the latest version, and you would expect that to be patched first. Serna told us that they chose to patch the most commonly used versions first. Which we find strange, users who keep the software up to date are now actually punished, they have to wait longer for the patch. Serna hinted that he can follow our reasoning, but that the development team set priorities and this choice was made. Furthermore, he emphasises once again that the temporary mitigation works on all versions, both the latest and old versions that are even no longer supported.
What lessons did Citrix learn from this?
What did Citrix learn from this? Would they make different choices in the future? Serna said that they still stand behind their procedures and the way they dealt with this. This procedure is also more or less an industry standard. In most cases, the procedure works just fine.
Serna added that he would like security researchers to be reasonable and that they are aware of the impact their actions have. He does not comment on specific security companies, but it is clear that “whitehat hackers” sometimes have their own agenda.
Citrix is going to investigate how they can make sure these kinds of errors don’t end up in the application code in the first place. That security has even more priority when developing applications. The fact is, however, and Serna immediately acknowledged this; preventing errors in software is impossible.
Hundreds or even thousands of people often work on these kinds of large applications. And where people work, mistakes are made. No matter how well companies try to control and optimise this. We can fill the website daily with news about security updates if we want.
Citrix has also put damage control high on the agenda
Citrix has been trying to limit the damage as much as possible over the past few weeks. Serna told us Citrix sales and customer success employees have been contacting customers to point out the security vulnerability and the temporary mitigation. Also, Citrix partners have helped where they could to inform customers.
The company has also introduced a tool that customers can use to analyse their Citrix servers. This tool is developed in cooperation with Forensic researcher FireEye. The tool can scan Citrix servers and recognise whether the security vulnerability was abused and whether known malware or ransomware may have been left behind. Citrix states that the tool is not able to identify all the consequences, so if the tool indicates that the vulnerability has been abused, it is wise to call in a forensic security company.
Whitehat hackers looking for attention and marketing for business
We have been tipped off before by software companies that the practices of some whitehat hackers are not always neat. Companies prefer not to talk about this, and Citrix also did not want to comment on the three security companies that have come forward. But you can ask yourself how happy Citrix actually has been with how the security researchers operated in this case.
The practices of malicious hackers are not good, but that also seems to be true for some whitehat hackers. These well-intentioned hackers can often be divided into two camps. Those who are looking for financial compensation they mainly choose software from companies that pay well for finding security vulnerabilities. Companies such as Facebook, Google and Microsoft pay large sums of money for critical leaks.
On the other hand, you also have security researchers who work for a company that sells security solutions and services. By finding and publishing security vulnerabilities, they get media attention, which is good for the company. Media attention is greater when there is a critical leak and especially when the solution is not immediately available. Security leaks that can be fixed immediately only generate minimal attention. Offering a waiting period of 90 days before publishing security vulnerabilities is good for security, but not for the attention that security firms are looking for. This might have been one of the root causes in this case.
24 hours ago
