GitHub has provided more details about a major outage that affected multiple services on August 17. A capacity issue at a U.S. data center escalated into an outage lasting nearly eight hours due to misconfigured autoscaling and aggressive retry mechanisms. This caused a sharp spike in requests, particularly for Copilot.
The issues began at 1:28 p.m. UTC at GitHub’s Central US data center. A traffic spike there saturated the load balancers. The initial problems arose with an Istio sidecar that reached its limit for concurrent connections. This component did not scale sufficiently because the configured autoscaling policy was based on the main service’s capacity rather than the sidecar’s limits.
The problem then spread to other parts of the infrastructure. Ultimately, four HAProxy nodes reached their maximum capacity. Because these systems were part of the authentication path, delays and errors occurred across various GitHub services.
Error rates as high as 50 percent
Issues, Pull Requests, APIs, Actions, and Copilot were affected. At its peak, the error rate for web and API traffic hovered around 20 percent. When downloading archives and raw content, this rose to about 50 percent. SAML and OIDC authentication, SCIM, and Team Sync also experienced issues.
GitHub rerouted some of the traffic from Central US to Northern Virginia. This initially provided some relief but did not resolve the underlying issue. Automatic retries actually ended up further increasing the load. Requests that were not processed or were processed too slowly were resent, forcing the already overloaded infrastructure to handle even more traffic.
Temporarily shutting down the affected HAProxy nodes ultimately led to a rapid recovery of a large portion of the services. Most components were functioning normally again by 4:36 p.m. UTC. GitHub Actions continued to experience issues until approximately 6:03 p.m.
Copilot traffic increases tenfold
The issues persisted longer for Copilot. An existing bug in VS Code’s retry behavior amplified traffic to the Copilot Token Service by approximately a factor of ten. A failed token request could trigger multiple new requests and subsequently get stuck in a retry loop.
As a result, traffic to the Token Service surged from the normal 7,000 to 9,000 requests per second to 70,000 to 100,000 requests per second. GitHub subsequently reduced the number of retries within its gateway and temporarily blocked certain token requests. Traffic was then gradually allowed back in on a per-location basis. The Copilot Token Service was fully restored at 9:02 p.m. UTC.
According to GitHub, the recovery was further complicated by scraping attacks on the codeload endpoints.
In response to the incident, GitHub will adjust the autoscaling of Istio components. It will also review retry and backoff mechanisms in gateways and clients. Additionally, the company plans to address VS Code retry behavior and improve monitoring of load balancer capacity and regional failover.
More availability issues
This outage is not an isolated incident. The Register points out that GitHub has struggled with availability issues more frequently in recent months. In early August, Actions and Pages, among other services, were affected. In May, an outage affecting Actions even caused some developers to mistakenly receive notifications that their accounts had been suspended. GitHub had previously acknowledged that the platform’s reliability needs improvement.
This series of incidents coincides with the emergence of new alternatives to GitHub. CloudBees CEO Moritz Plassnig expects the code hosting market to become less centered around a single platform, pointing to Cursor and OpenAI, among others. This week, Cursor introduced Origin Code Hosting, its own Git-based hosting service that can compete directly with GitHub.