The massive global outage that plunged Facebook, its Instagram and WhatsApp platforms and many people who rely heavily on these services — including Facebook’s own workforce — into chaos Monday has dissipated.
Facebook said late Monday that it's been working to restore access to its services and is “happy to report they are coming back online now." The company apologized and thanked its users for bearing with it. But fixing it wasn't as simple as flipping a proverbial switch. For some users, WhatsApp was working for a time, then not. For others, Instagram was working but not Facebook, and so on.
Facebook's outage began around 11:40 a.m. ET and was still not fixed more than six hours later.
Shortly after 10 p.m. Eastern, Santosh Janardhan, Facebook's Vice President of Infrastructure, said the underlying cause of Monday's outage impacted many of the company's internal tools and systems, which complicated work to diagnose and fix the issue.
"Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt," Janardhan wrote in a blog post.
"We want to make clear at this time we believe the root cause of this outage was a faulty configuration change. We also have no evidence that user data was compromised as a result of this downtime," Janardhan added.
Facebook was already in the throes of a separate major crisis after whistleblower Frances Haugen, a former Facebook product manager, provided The Wall Street Journal with internal documents that exposed the company’s awareness of harms caused by its products and decisions. Haugen went public on CBS’s “60 Minutes” program Sunday and is scheduled to testify before a Senate subcommittee Tuesday.
Haugen had also anonymously filed complaints with federal law enforcement alleging Facebook’s own research shows how it magnifies hate and misinformation and leads to increased polarization. It also showed that the company was aware that Instagram can harm teenage girls’ mental health.
The Journal’s stories, called “The Facebook Files,” painted a picture of a company focused on growth and its own interests over the public good. Facebook has tried to play down their impact. Nick Clegg, the company’s vice president of policy and public affairs, wrote to Facebook employees in a memo Friday that “social media has had a big impact on society in recent years, and Facebook is often a place where much of this debate plays out.”
The outage didn't exactly bolster Facebook's argument that its size and clout provide important benefits for the world. London-based internet monitoring firm Netblocks noted that the company's plans to integrate the technology behind its platforms — announced in 2019 — had raised concerns about the risks of such a move. While such centralization “gives the company a unified view of users’ internet usage habits,” Netblocks said, it also makes the services vulnerable to single points of failure.
“This is epic,” said Doug Madory, director of internet analysis for Kentik Inc, a network monitoring and intelligence company. The last major internet outage, which knocked many of the world’s top websites offline in June, lasted less than an hour. The stricken content-delivery company in that case, Fastly, blamed a software bug triggered by a customer who changed a setting.
For hours, Facebook’s only public comment was a tweet in which it acknowledged that “some people are having trouble accessing (the) Facebook app” and said it was working on restoring access. Regarding the internal failures, Instagram head Adam Mosseri tweeted that it feels like a “snow day.”
Mike Schroepfer, Facebook’s outgoing chief technology officer, later tweeted “sincere apologies” to everyone impacted by the outage. He blamed networking issues and said teams were “working as fast as possible to debug and restore as fast as possible.”
There was no evidence as of Monday afternoon that malicious activity was involved. Matthew Prince, CEO of the internet infrastructure provider Cloudflare, tweeted that “nothing we’re seeing related to the Facebook services outage suggests it was an attack.” Prince said the most likely explanation was that Facebook mistakenly knocked itself off the internet during maintenance.
Facebook did not respond to messages for comment about the attack or the possibility of malicious activity.
While much of Facebook's workforce is still working remotely, there were reports that employees at work on the company's Menlo Park, California, campus had trouble entering buildings because the outage had rendered their security badges useless.
But the impact was far worse for multitudes of Facebook's nearly 3 billion users, showing just how much the world has come to rely on it and its properties — to run businesses, connect with online communities, log on to multiple other websites and even order food.
It also showed that despite the presence of Twitter, Telegram, Signal, TikTok, Snapchat and a bevy of other platforms, nothing can easily replace the social network that over the past 17 years has effectively evolved into critical infrastructure. The outage came the same day Facebook asked a federal judge that a revised antitrust complaint against it by the Federal Trade Commission be dismissed because it faces vigorous competition from other services.
There are certainly other online services for posting selfies, connecting with fans or reaching out to elected officials, But those who rely on Facebook to run their business or communicate with friends and family in far-flung places saw this as little consolation.
Kendall Ross, owner of a knitwear brand called Knit That in Oklahoma City, said he has 32,000 followers on his Instagram business page @id.knit.that. Almost all of his website traffic comes directly from Instagram. He posted a product photo about an hour before Instagram went out. He said he tends to sell about two hand-knit pieces after posting a product photo for about $300 to $400.
“The outage today is frustrating financially,” he said. “It’s also a huge awakening that social media controls so much of my success in business.”
The cause of the outage remains unclear. Madory said Facebook appears to have deleted basic data that tells the rest of the internet how to communicate with its properties. Such data is part of the internet’s Domain Name System, a central component that directs its traffic. Without Facebook broadcasting its location on the public internet, apps and web addresses simple could not locate it.
So many people are reliant on Facebook, WhatsApp or Instagram as primary modes of communication that losing access for so long can make them vulnerable to criminals taking advantage of the outage, said Rachel Tobac, a hacker and CEO of SocialProof Security.
“They don’t know how to contact the people in their lives without it,” she said. “They’re more susceptible to social engineering because they’re so desperate to communicate.” Tobac said during previous outages, some people have received emails promising to restore their social media account by clicking on a malicious link that can expose their personal data.
Jake Williams, chief technical officer of the cybersecurity firm BreachQuest, said that while foul play cannot be completely ruled out, chances were good that the outage is “an operational issue” caused by human error.
“What it boils down to: running a LARGE, even by Internet standards, distributed system is very hard, even for the very best,” tweeted Columbia University computer scientist Steven Bellovin.
Twitter, meanwhile, chimed in from the company’s main account on its service, posting “hello literally everyone” as jokes and memes about the Facebook outage flooded the platform. Later, as an unverified screenshot suggesting that the facebook.com address was for sale circulated, Twitter CEO Jack Dorsey tweeted, “how much?”