Locked out and totally down: Facebook’s scramble to fix a massive outage

Engineers had to visit a California data center in person

Locked out and totally down: Facebook’s scramble to fix a massive outage0

A prolonged, global outage of Facebook’s apps sent the company’s engineers scrambling to fix the issue at one of its data centers in California, according to two people familiar with the situation.

The outage, which began around 11:40AM ET on Monday, brought down all of Facebook’s apps — including Instagram and WhatsApp — globally, affecting billions of users and millions of advertisers. Inside Facebook, the outage also broke nearly all of the internal systems that employees use to communicate and work. As of 6PM ET, it appears that most of the services are back online.

Several employees told The Verge they resorted to talking through their work-provided Outlook email accounts since Facebook mainly runs on an internal version of the social network that is currently not accessible. While employees could email each other, they couldn’t send or receive emails from external addresses.

Since Facebook requires employees to log in with their work accounts to access tools such as Google Docs and Zoom, those services also weren’t working, leading some employees to use alternative services like Apple’s FaceTime and Discord. Employees who were already authenticated with non-Facebook tools like Google Docs before the outage began still had access.

Facebook engineers were sent to one of its main US data centers in California to restore service, meaning the fix couldn’t be done remotely. Further complicating matters, the outage temporarily broke the ability for some employees to access company buildings and conference rooms with their badges, according to The New York Times, which first reported that engineers had been dispatched to the data center.

In an email to employees sent shortly after service was restored, CTO Mike Schroepfer said the issue was “affecting our networking backbone that connects all our data centers together.”

“If you are not actively working on the recovery, please be patient and don’t rush to reload everything to prevent slowing down the bring up” of the network, he cautioned in the memo that was seen by The Verge.

Facebook has yet to detail the exact cause of the outage, though outside experts are already saying it was due to an issue with the networking technology BGP, or Border Gateway Protocol.

Update October 4th, 6:33PM ET: Noted that the outage is ending as Facebook and its other services are coming back online.

Update October 4th, 8:05PM ET: Added more information about the outage that was shared with Facebook employees.

Enable registration in settings - general