A sensor that expected 21 inputs and only got 20 was the reason some 8.5 million Windows systems crashed on Friday, July 19. This had major consequences for airlines, hospitals, hotels and supermarkets worldwide, to name just a few affected industries. Crowdstrike’s final Root Cause Analysis (RCA) goes into further detail.
The analysis states that the company rolled out a new version of an existing sensor functionality for the Falcon security platform last February. This version included an IPC template type that allowed it to quickly pass along insights about new attack methods that abuse the existing operation of Windows components. The idea was that new attack methods could be cut off very quickly through so-called Rapid Response Content releases.
These updates appear much more frequently than Sensor Content, several times a month, and provide so-called ‘Channel Files’ with updates. In this way, the sensor detects new specific forms of malicious behaviour. These updates allow CrowdStrike to respond to the most current threats.
Out-of-bounds memory read
That was working properly until an update numbered 7.11 in July caused only 20 of the expected 21 input parameter fields to be filled. The 21st was normally reserved for some ‘wildcard,’ which means, among other things, that inspection is not required. The latest update lacked such wildcard-matching criteria, leading to a logical error. Those criteria were still present in all previous test phases but were missing in the final rollout.
Tip: Crowdstrike warns against fake recovery manual
The attempt to read the 21st value led to an out-of-bounds memory read, resulting in a system crash. A patch that controls the number of input fields at compile time has been available for some time now. On August 9, additional hotfixes will also become available that always check whether the number of input fields matches the supplied inputs. That should prevent out-of-bounds access. Also, the hotfix ensures the required number of input fields for the template types will always be 21.
To prevent such crashes from now on, Crowdstrike will build in additional test phases before rollout to production. Also, such Rapid Response updates will come online gradually from now on instead of going to all users at once. Furthermore, customers will now have more control over receiving and deploying Rapid Response Content. Further, the analysis reports that Crowdstrike has hired two outside security firms to scrutinize Falcon’s code.
Lawyers of Delta and Microsoft cross swords
After the massive outage in July, Delta, the world’s largest airline, claimed that CrowdStrike was responsible for delayed and cancelled flights. Although CrowdStrike did apologize, it rejected that claim. Delta also allegedly refused (free) assistance. There is no official charge or lawsuit yet, but Delta is looking at whether CrowdStrike and Microsoft are somehow responsible.
Microsoft also believes that Delta has much to blame itself, according to a letter to the lawyer Delta hired, David Boies. Boies has fought the tech giant in the past.
The letter from Microsoft’s own lawyer states bluntly that “our preliminary review suggests that Delta, unlike its competitors, apparently has not modernized its IT infrastructure, either for the benefit of its customers or for its pilots and flight attendants.”
Also read: CrowdStrike licks its wounds after catastrophic update