Microsoft has disclosed details of a large-scale Outlook failure, due to which company employees, government agencies and ordinary users lost access to email for almost a day. The problem affected more than 80% of Exchange Online servers and spread to several countries due to failover mechanisms.

Image Source: Simon Ray/Unsplash
According to The Verge, the outage began on November 12 around 9:00 am. Exchange Online servers in Switzerland were the first to stop responding, followed by problems in Mexico and other countries. Failover systems redirected mail traffic to other servers, but instead of containing the problem, they allowed it to spread. Within a few hours, more than 80% of Exchange servers stopped responding. Microsoft has assigned the incident the highest level of criticality Sev0 and involved hundreds of engineers in the investigation.
At first, experts studied trace logs, crash reports and core dumps. Some servers went into maintenance mode and stopped responding due to excessive CPU load. Attempts to restart them did not produce results: after restoration of operation, the equipment immediately received a stream of accumulated requests and failed again. In some cases, a reboot cycle occurred. Additional difficulties were created by the fact that standard procedures for rolling back changes did not allow returning the servers to a working state.
As a result, Microsoft discovered a hidden error in the Microsoft 365 network infrastructure. It had been present on the network gateway servers since July, but did not manifest itself in any way until the November incident. According to the company’s preliminary report, the problem was related to a lock conflict within one of the internal Windows processes. Under certain load conditions, this caused the servers to stop responding.
Regular versions of Windows are not affected: the problem was caused by a patch prepared specifically for Microsoft 365 network gateways. According to Charles Lamanna, head of Copilot, Agents and Platform at Microsoft, the problem was more pronounced on machines with more cores and newer hardware. At the same time, in the company’s testing laboratories, load tests were mainly carried out on less powerful systems with a small number of cores, so the defect could not be identified in advance.
The failure was the result of a combination of several factors. 15 hours before the problems with Outlook began, Microsoft Teams switched from an alternative network gateway to the main one serving Exchange Online. This increased the load on the infrastructure and changed the nature of network traffic. At the same time, about 14% of the main gateway’s power remained unavailable due to another fault that had occurred the day before. Combined with a hidden bug, this led to a massive server failure.
Microsoft also checked the possibility of a cyber attack, but logs and crash reports allowed it to be ruled out. To coordinate the restoration work, engineers held continuous meetings: one such call involved up to 1,000 people simultaneously.
It took almost 24 hours to stabilize the Exchange Online servers. Microsoft admitted that standard recovery mechanisms were not designed for such a large-scale failure, and standard rollback procedures did not help quickly eliminate the defect that had been present in the infrastructure for several months. The company is preparing a final report on what happened and promises to publish it later.
If you notice an error, select it with the mouse and press CTRL+ENTER.


