Released by GitHub Management Report Regarding a previous outage that lasted eight hours. The cause of the problem is related to incorrect operation of the load balancer and a Visual Studio Code error, resulting in a series of duplicate requests.
Photo credit: Rubaitul Azad/unsplash.com
Problems accessing GitHub began at 16:28 Moscow time on August 17; they were not completely overcome until 0:15 Moscow time on August 18. The incident lasted for 7 hours and 47 minutes, causing an increase in the number of errors across GitHub services. The main reason for the failure was that the load balancer network at the company’s US headquarters was overloaded – this was triggered by reaching the Istio sidecar concurrency limit. When these limits are reached, the system should increase throughput, but due to incorrect policy configuration, this does not happen because only the host service is monitored, not the load balancer status. The problem quickly worsened as clients started sending repeated requests, causing the balancer to completely deactivate.
Engineers resolved the issue by temporarily reducing the number of gateway retries by changing the code and configuring the load balancer to deny incoming requests to the Copilot token service with error code 403. The outage also exposed a long-standing bug in VS Code – the program started sending duplicate requests, resulting in a tenfold increase in traffic and delays in restoring the Copilot token service. Most services resumed at 19:36 Moscow time, Actions resumed at 18:03 Moscow time, and the Copilot token service only started working on August 18 at 0:02 Moscow time.
The company promised to fix bugs in the autoscaling policy, modify retry limits, review Istio concurrency settings and fix behavioral issues with VS Code that were causing increased traffic to the Copilot token service.
If you find an error, select it with your mouse and press CTRL+ENTER.