Google has confirmed that an incorrect network configuration was the cause of the outage that affected Google Cloud VMware Engine (GCVE) customers with a stretched cluster on Tuesday. The error caused communication between different availability zones to fail, meaning that while virtual machines continued to run, in many cases they were no longer accessible.
The outage began on July 14 at 10:00 a.m. Pacific Time and was not fully resolved until nearly twelve hours later. According to Google, customers in Frankfurt, Sydney, and Melbourne, among other locations, were affected. Only environments using stretched clusters were impacted; standard GCVE deployments were unaffected.
A stretched cluster is designed to increase the availability of VMware workloads by spreading a cluster across two physical locations or availability zones. If one location goes down, the other must be able to take over service without interruption. It was precisely this interconnection that proved to be the weak link during this incident.
Diagnosis shifted throughout the day
In the first few hours, Google reported only that there was a network issue between the zones of a stretched cluster. The company emphasized that compute and storage services continued to function and that the virtual machines themselves had not gone down. According to Google, the problem lay in the accessibility of those systems.
Later in the day, the analysis became more specific. Google reported that the connections between the cluster zones and the so-called witness appliance had failed. There were also issues with Border Gateway Protocol (BGP) sessions, preventing the various cluster components from synchronizing their status. In that scenario, there is a risk that virtual machines become isolated and no longer have writable storage available.
Rollback resolved the problem
A few hours later, Google identified a recent network configuration update as the likely cause. In its final incident analysis, the company confirmed that this change was indeed responsible for the disruption in communication between the zones.
The solution involved rolling back the configuration to the last known working version. After this rollback, the connection between the cluster locations was restored, and affected customers were able to use their environments normally again. Google asks customers who are still experiencing residual issues to contact support.
The official incident analysis thus clarifies what was previously unclear. While Google had previously referred to an underlying network issue during the outage, the company now confirms that a configuration change it made was the direct cause of the prolonged disruption affecting GCVE stretched clusters.