Why Facebook, Instagram and WhatsApp all went down today

On Facebook, Instagram, WhatsApp and Oculus disconnection hit all corners of Mark Zuckerberg’s empire offline Monday. It’s a blackout on social media that can most beneficially be described as “thorough” and is likely to be particularly hard to correct.

Facebook itself has not confirmed the root cause of its problems, but there are plenty of clues on the internet. The company’s family of apps effectively fell off the Internet at. 11:40 ET, according to when its Domain Name System records became unavailable. DNS is often referred to as the Internet directory; it’s what translates the hostnames you type into a URL tab – like facebook.com – into the IP addresses where these sites reside.

DNS crashes are common enough, and when in doubt, it’s the reason a given site has crashed. They can happen for all sorts of skewed technical reasons, often related to configuration issues, and can be relatively straightforward to solve. In this case, however, something more serious seems to be going on.

“Facebook’s disruption appears to be caused by DNS; but it’s just a symptom of the problem, ”said Troy Mursch, chief researcher at the cyber-threat intelligence firm Bad Packets. The basic problem, Mursch says – and other experts agree – is that Facebook has withdrawn the so-called Border Gateway Protocol route, which contains the IP addresses of its DNS name servers. If DNS is the Internet directory, BGP is its navigation system; it determines what route data it takes when it runs the information super garden.

“You can think of it as a gaming phone,” but instead of people playing, it’s smaller networks letting each other know how to reach them, says Angelique Medina, product marketing director at network monitoring firm Cisco ThousandEyes. “They are announcing this route to their neighbor, and their neighbor will convey it to their peers.”

It’s a lot of jargon, but easy to say: Facebook has fallen off the map of the internet. Are you trying to ping these IPs right now? “The packages end up in a black hole,” Mursch says.

A map shows where Facebook is inaccessible due to errors in DNS resolution – pretty much it is everywhere, all at once.

Greetings from Cisco Thousand Eyes

The obvious and still unresolved question is why these BGP routes disappeared in the first place. It is not a common disorder, especially on this scale or in this duration. Facebook has not said anything beyond a tweet that it is “working to get things back to normal as soon as possible.” But the Internet infrastructure experts who spoke to WIRED all suggested that the likely answer was a misconfiguration on Facebook’s part. “It looks like Facebook has done something about their routers, the ones that connect the Facebook network to the rest of the Internet,” said John Graham-Cumming, CTO of Internet infrastructure company Cloudflare, who stressed that he did not know the details of what happened. After all, he says, the Internet is essentially a network of networks, each advertising their presence to the other. For once, Facebook has stopped advertising.


Leave a Comment