Engineering

How Facebook kept the lights on when the world went remote

November 16, 2020

Every business has to be prepared to handle its most difficult stress test: dinner service at a restaurant, tax season for accountants, or the holiday season for retail. For companies like Facebook that are in the business of connecting people, it’s New Year’s Eve — a single-day global celebration accompanied by a huge spike in messaging, photo uploads, and social sharing that’s unlike anything else we see the rest of the year.

In March 2020, we saw the beginnings of a traffic spike that would dwarf New Year's Eve several times over — and it lasted for months. It came as the world entered a crisis without precedent: Much of the physical world simply shut down and was almost immediately supplanted by the internet. Usage of our products in many countries around the world increased by up to 40 percent week over week, as populations adjusted to lockdowns and shifted life online.

Passing this stress test would have been impressive. But what we saw in those early months of the crisis was a different story. In confronting truly enormous challenges, our engineering team did more than overcome them: They came together to drive unprecedented efficiency improvements and make our infrastructure more resilient and efficient, and they built entirely new products and features to power a remote global workforce.

Watch the videos from our event, Keeping the Lights On @Scale, to hear more about the challenges, breakthroughs, and lessons learned by our engineering teams during the early months of the COVID-19 crisis. 

For some of our personal reflections and stories from those months, flip through the slideshow below. 

The surge began almost immediately after people across the country and the world started working from home in early March. Engineers on our infrastructure teams saw demand on our servers rise in ways they had never experienced before — the equivalent of months' worth of steady global growth in demand happening within days. 

Our teams had begun planning for scenarios like this when Italy went into lockdown in late February, and by the beginning of March they realized we needed a centralized response not just for keeping our users online but also for keeping our own teams safe. On March 6, a group met at our headquarters in Menlo Park to discuss how our own operations would handle an all-remote workforce. They finished the meeting, and within hours every Facebook office in the world was shut down.

By the end of April, the volume of messages sent on our Messenger app in countries hit hard by the virus had risen by 50 percent; video calling had doubled. In Italy, one of the first countries to enter a national lockdown, the volume of group video chats rose by 1,000 percent in a single month. By June, live video broadcasts from pages had doubled from the year prior.  

Across the world, new ways of working, socializing, and entertaining were emerging, and the things that we needed to survive and thrive as a company were also what our users demanded of us. And so after a few initial weeks of scrambling to keep our business running and services online amid crushing demand and a reordered economy, we moved to the phase we’re still in to this day: building products and services for a fundamentally changed world.

Video calling and remote collaboration has been a central part of this. It began with the needs of our own staff — tens of thousands of people spread across the globe, whose demand for videoconferencing quadrupled overnight and who suddenly needed vastly more access to our internal networks. And it wasn’t just existing staff: Between April and September, Facebook hired more than 8,000 new employees across the world, in a process that had shifted entirely online. 

The sudden need among our staff to massively expand their virtual communications was also shared by the users of our apps, who quickly began making more than a billion minutes of WhatsApp video calls each day and using Messenger in ways we hadn’t seen before. 

In each of our apps, we saw people communicating and sharing in far greater volumes, and needing new options for their new reality. Our engineers responded with incredible speed and focus: Within six weeks of the WHO declaring a pandemic in early March, we’d launched entirely new products designed to help people stay connected in this new, socially distant reality. Teams raced to prioritize these new products, like Messenger Rooms for group video chat and Facebook Shops for small businesses suddenly needing to pivot to an online-only world. Products that would normally take up to a year to build and launch were completed in just a couple of months, as most of our engineers focused on the products most needed by people working, learning, and socializing from home. 

We saw similar dynamics in Workplace, our business collaboration product, used by more than 5 million paid users around the world, and on Portal, our dedicated video calling hardware. And when Quest 2, our new virtual reality headset, became available for preorders in September, demand was through the roof. 

In each of these spaces, our engineers are still learning about the new ways people are communicating and collaborating. They’re still learning how they can work together to keep an extraordinarily complicated operation running under the weight of an unprecedented surge in demand, and they’re doing it all from bedrooms and dining tables. And from these learnings — the daily experiences of a new world, shared by our staff and our users — we’re seeing glimpses of the future of the internet.

To learn more about the work we did in the early months of the COVID-19 pandemic, join the Keeping the Lights On @Scale series, which starts on November 24.

We're hiring engineers!

Help us build infrastructure and solve big challenges at scale

Engineering

Meta’s engineering teams create the infrastructure and systems that underpin our apps and services, connecting more than 2 billion people.