A long-running, global outage of Facebook apps sent the company’s engineers to fix the problem at one of its data centers in California, according to two people who know the situation.
The interruption, which began around 6 p.m. At 11:40 a.m. Monday, all of Facebook’s apps – including Instagram and WhatsApp – crashed globally, affecting billions of users and millions of advertisers. Inside Facebook, the outage also broke almost all of the internal systems that employees use to communicate and work. From 18.00 ET it seems that most of the services are back online.
Several employees told The edge they resorted to talking through their work-delivered Outlook email accounts, as Facebook mainly runs on an internal version of the social network that is not currently available. While employees could email each other, they could not send or receive emails from external addresses.
Since Facebook requires employees to sign in with their work accounts to access tools like Google Docs and Zoom, those services did not work either, prompting some employees to use alternative services like Apple’s FaceTime and Discord. Employees who were already authenticated with non-Facebook tools like Google Docs before the outage began still had access.
Facebook engineers were sent to one of its most important U.S. data centers in California to restore service, meaning the fix could not be made remotely. Further complicating matters, the interruption temporarily disrupted the ability of some employees to access the company’s buildings and conference rooms with their badges, according to New York Times, who first reported that engineers had been sent to the data center.
In an email to employees sent shortly after the service was restored, CTO Mike Schroepfer said the problem “affects our network strain that connects all of our data centers together”. ”
“If you are not actively working on the recovery, be patient and do not rush to reload everything to prevent slowing down the recording” of the network, he warned in the note seen by The edge.
Facebook has not yet detailed the exact reason for the outage, although external experts already say it was due to a problem with the BGP or Border Gateway Protocol network technology.
Updated October 4, 18:33 ET: Noticed that the outage ends when Facebook and its other services return online.
Updated October 4, 20:05 ET: Added more information about the outage that was shared with Facebook employees.
Updated October 4, 21:06 ET: Added the confirmed report that an angle grinder was used to access server cages.
Correction 4 October, 21:25 ET: An earlier version of this story included a confirmed report that Facebook used an angle grinder to access server cages. The reporter has since withdrawn the confirmation and we have removed the claim from this story.