Messaging, voice and video calling have surged in recent months during the COVID-19 pandemic as people around the world check in with family, friends and colleagues. Audiences for newscasts and government briefings have also ballooned as the public seeks updates on the outbreak, travel guidance and personal hygiene advice to protect themselves from getting sick.

While there is no shortage of information, not everyone can access it. It needs to be available to the hundreds of millions of people in the world who are deaf or hard of hearing. An estimated 70 million people are deaf and 300 million are hard of hearing. While Facebook provides automatic closed captioning for on-demand videos in 16 languages, and just announced similar capabilities for Instagram IGTVaccess to live, real-time news and information is still a need to be met. 

Facebook AI researchers and engineers have now made live video content more accessible by enabling automatic closed captions for Facebook Live and Workplace Live. Already, six languages are supported: English, Spanish, Portuguese, Italian, German and French. Facebook Live automatic captions are helping ensure that millions of viewers across the world – whether they are deaf or hard of hearing, or are just watching where audio is not available – get the message. And, as workplace policies evolve, automatic captioning has become essential for employers to keep their staff and customers informed.

The speed and scale of this AI-powered technology was only possible thanks to advances Facebook AI has made in automated speech recognition (ASR) over the past few years.

Laying out the challenge 

Although automated caption technology, which predicts a sequence of words from a raw audio signal, has been around since the late 2000s, it is still an exceptionally difficult task. In the type of conversational speech that is present in live streams, people don’t always naturally speak clearly or wait their turn to speak. Unpredictable background noise, the large variety of accents and dialects, and the wide range of tones that influence human speech, make ASR even harder. 

The system also needs to learn to recognize hundreds of millions of different words across many languages, including uncommon names and jargon. An open domain task like this is very different from, and much more complex than, more constrained ASR tasks such as automated customer service calls where the system only needs to consider a relatively small set of possibilities.

Conventional ASR systems are made up of three components: an acoustic model that predicts phonemes from short segments of audio, a pronunciation lexicon, which describes how the phonemes are combined to form the words of a given language, and a language model that captures the relationships among those words, e.g. which words are the most common and which words are likely to appear together.

A pivotal early discovery by the Facebook AI team was that the phonetic pronunciation lexicon could be eliminated, and acoustic models could be trained to directly predict the graphemes (or characters) of a word with better accuracy for end-to-end systems at first, and later also confirmed for hybrid systems. This greatly simplified training and deployment of these ASR models across different languages.

The rapid spread of the COVID-19 pandemic caused a spike in both the supply and demand of public health information. Several local and state governments, accustomed to holding live press conferences, didn’t have the resources, staff or technology to record, stream, and caption their live events, so they turned to Facebook Live. 

People around the world were also tuning into newscasts and conferences streaming on Facebook Live, and watching for much longer periods of time than usual. In fact, the number of Facebook Live broadcasts from Pages doubled in June 2020 compared to the same time last year. That incredible amount of traffic puts enormous stress on any ASR system.

To handle these elevated spikes in traffic, Facebook’s ASR models needed to get a lot faster in production to avoid falling behind. Recent research has shown that convolutional encoders trained with the CTC loss function could increase efficiency during inference for streaming use cases, while RNN Transducer models consistently yielded the best accuracy despite being the most compact. In non-streaming use cases, (i.e. when the entire video is available to the model for decoding) we have found that Transformer encoders can produce ASR models that are both very fast and the most accurate.

Facebook engineers were able to deploy these model variations with a number of infrastructure optimizations, which enabled Facebook to serve all the additional video traffic and resulted in machine savings despite the increased load. Models were trained using PyTorch which enabled quick iterations on ideas and deployments to production.

“Improving speed without compromising on accuracy is the cherry on top,” says Yatharth Saraf, an Engineering Manager at Facebook.

Julian Chan, a Facebook AI software engineer explains that the system is also capable of adapting to new words such as “COVID,” which is a hot topic of discussion.“ It can easily learn a new word and predict where it will occur,” he explains. “This was largely made possible using text data from public Facebook posts to train the system.” 

What’s next

The training data our system learned from included many different types of speech, but it’s far from perfect, especially when it comes to different accents. However, it can be difficult or even impossible to collect sufficient training data of every type, so researchers are exploring methods to improve and adapt models by having them also learn from vast amounts of unlabeled audio.

This story has been updated to reflect feedback from the National Association of the Deaf.

Acknowledgements 

We'd like to extend special thanks to Xiaohui Zhang, Duc Le, Frank Zhang, Jun Liu, Si Chen, Chunxi Liu, Jiedan Zhu, Yongqiang Wang, Vineel Pratap, Jiatong Zhou, Julian Chan, Kritika Singh, Kjell Schubert, Yutong Pang, Shahzad Bhatti, Jeff Glick, Andres Alvarado, Mark Chou, Abdelrahman Mohamed, Awni Hannun, Ronan Collobert, Mike Seltzer, and Geoffrey Zweig