5 min Security

Maintenance, faulty judgment, and a bug from July took Microsoft 365 offline for a day

Maintenance, faulty judgment, and a bug from July took Microsoft 365 offline for a day

It’s rare for a cloud service outage to last longer than a few hours. It’s also uncommon for most of Microsoft’s online suite to be inaccessible. The fact that it happened on Friday the 13th was due to a combination of a bug from July, maintenance work, incorrect assumptions, and an overlooked issue: a lack of control over its own infrastructure.

In a sense, the Microsoft 365 outage from Thursday evening through Friday evening is reminiscent of the CrowdStrike outage in July 2024. The cause then and now was a software bug that went unnoticed for some time because conditions were favorable. Subsequently, the normal operation of the company’s own network caused the bug to spread, without the affected company knowing exactly what was going on.

A bug from July

First, the key points: a “latent defect” in Windows servers led to lock contention on systems that differed from the test environment. This bug was introduced unnoticed in July while resolving another issue. In lock contention, multiple threads compete for the same locked resource. They must wait for one another, causing the CPU to spend its time waiting instead of working. Microsoft states that this problem worsens as modern processors have more cores.

It is reminiscent of a race condition, in which unsynchronized access causes data to be accessed or modified simultaneously, often resulting in a crash or the failure of a service or feature. In that case, uncontrolled concurrency is the culprit, whereas lock contention arises precisely from the enforcement of exclusive access. Requests then pile up, causing the CPU and system memory to stop responding.

Microsoft 365 uses two separate network gateways, logically to prevent a single issue from bringing down the entire system. However, due to an unrelated issue, Teams had been switched from the alternate gateway to the primary one, where Exchange Online also runs. In addition, 14 percent of server capacity was temporarily unavailable due to another, equally unrelated issue from the previous day.

Subsequently, Friday’s traffic, particularly because it coincided with the peak in activity in the United States, caused a sharp increase in routing to alternative regions. Normally, this helps spread out the load and keep servers running smoothly, but in this case, it only caused the bug from July to strike more frequently. At its worst, up to 80 percent of the entire Microsoft 365 server fleet was unreachable.

23 hours and 50 minutes

Aside from the points Microsoft shares in the Post-Incident Report, which is unhelpfully tucked away in the Microsoft 365 admin center, there’s an underreported issue: simply the duration of the outage. It took Microsoft nearly a full day to go from the first meaningful detection of problems to a complete fix. For small and medium-sized businesses, that might be considered quick, but more is expected from Microsoft, especially given its massive investments in resilience.

Moreover, the rollout of the fix did not go smoothly. The restored servers were almost immediately overloaded by the backlog of traffic waiting to be processed. Microsoft therefore had to gradually bring the hardware back online to prevent continued outages. At a more fundamental level, the automated recovery service proved not to be designed to handle such a catastrophic situation, a glaring oversight.

Three mistakes, partly forgivable

The uptime figures for Q3 2026 are still pending (presumably later this month), but in the first two quarters of 2026, Microsoft reported availability of 99.526 percent (Q1) and 99.994 percent (Q2). The SLA for Microsoft 365 Business Premium guarantees 99.9 percent uptime per calendar month. Looking back over the past few years, we see that this threshold has always been met, except in Q1 2026. With a full day of downtime, Q4 2026 could set a new low, as this would result in a maximum availability of 96.7 percent (1 day is 3.3 percent of a quarter).

The errors are somewhat complex but could have been prevented. The fact that Microsoft tests on weaker hardware than what typically runs in the public cloud needs to be addressed. Modern chips have very different cache, core, and frequency characteristics than the hardware from a few years ago. On this scale, significant updates must be tested on precisely that hardware.

Less attributable to Microsoft is the unfortunate coincidence of maintenance and previous issues; such a combination doesn’t happen often. In theory, you can make your infrastructure resilient against a downright absurd series of problems, but at some point, those costs become impossible to justify to finance teams or executives. However, it does seem that recovery capabilities need to be better equipped to handle a “code red” situation like this one. It’s conceivable that a cloud could go completely down, whether due to malicious actors or bugs. The recovery should then have been able to rely on a phased restoration that, while perhaps not the fastest route to availability, would have fallen back on a safe, older configuration.

In that sense, it appears that Microsoft did not discover the actual cause, the bug from July, until quite late. The company has promised a more detailed report this week.

See also: CrowdStrike licks its wounds after catastrophic update (2024)