From vinyl to the cassette tape to the mp3, the dominant distribution means of popular music have evolved considerably over the last several decades. What’s remained constant is the importance of cover art — a visual signifier and shorthand for an album’s themes, motifs, and emotional weight. It’s often the first experience a fan has of an album, and it has a unique potential to bring them closer to the music. Recently, Facebook teamed up with pop rock band HAIM to bring the cover art for its album Women in Music Pt. III to life using the magic of augmented reality (AR) and the Spark AR platform.

Haim was set to tour in London, and Facebook was exploring ways to incorporate AR into the experience. Initial concepts included augmenting the kick drum or banners hung beside the stage, as well as the band’s merchandise. When the gigs were postponed as a result of COVID-19, the team zeroed in on the album itself.  

“The album cover has always been so integral to that first experience with a piece of music,” notes Facebook Creative Shop’s Ricardo Caetano. “I particularly always loved the large format LP — opening, unfolding, linking the artwork to the lyrics. As fans of music, art, and technology, we were really interested in exploring a collaboration where we could design a music experience leveraging AR as a bridge to interact with the cover. We wanted to let people play and customize the visuals where every interaction would affect the soundscape.”

No stranger to AR, the band’s label was enthusiastic. “We’ve gradually been working more and more AR activations into the fabric of our campaigns over the last year,” explains Polydor Records UK/Universal Music UK Digital Marketing Lead Ed Juniper. “Often it’s as simple as a face filter that ties in with the artwork or music video, but increasingly the technology is there to expand the world of an artist’s song or album with really innovative, creative augmented reality that fans can interact and engage with.”

Augmenting A/V

The concept in question was an augmented cover that would double as a music sampler, incorporating visuals and audio that were evocative of the album’s sonic identity.

“Audio is a really underappreciated part of the AR experience,” says Facebook Creative Shop’s Dan Moller. “And with our new technology coming online and deeper investment in the space, we wanted to push the audio to see what we could achieve.”

“Our internal sound design team produces a lot of original audio content for Instagram and other platforms, but this project gave us the valuable opportunity to design a creative reinterpretation of existing audio content,” adds Facebook Sound Design’s Matt Nichols. “We had a lot of fun exploring the wonderful music on this album and figuring out the most effective ways to excerpt and interpret it as an interactive Spark AR experience.”

While the simple route would have been to play a single loop of one of the album’s tracks, both Haim and the team at Facebook leaned toward a more experimental solution. “In our final audio design, we were able to use a collection of sounds we sourced from the album to create an ethereal, dreamy soundscape, capturing the sonic essence of the album in a creative way,” Nichols says.

The AR effect provides digital overlays, highlighting the track listing, doodles, and the band members’ signatures. Each interaction triggers audio and visuals, lending itself to a one-of-a-kind viewing experience every time. 

“The Haim project was really pioneering in how embedded the artist’s own music was in an effect,” says Juniper. “More than merely soundbedding the effect, the user could interact and play with a soundscape crafted from album stems. The level of interactivity with the music I imagine will only increase in the coming years, with the lines increasingly blurred between showcasing the artist’s music and equipping fans with the tools to have rich experiences and create something of their own in AR visuals and sound.”

Overcoming obstacles

While the finished product may be straightforward, the path to success was nuanced.

“There’s a huge opportunity here to make the effect consumer feel more present with an experience by using sound, and obviously that’s extremely important when designing promotional material for an album,” says Nichols. “When designing Spark AR experiences that relate to the world around us, our primary challenge is that we aren’t yet able to play sounds as if they’re actually being emitted from a particular location — we call this ‘spatialization,’ and it’s a common technique in other immersive media, most notably in video games. Even though we don’t yet have this technology in Spark AR, we’ve found a few creative solutions that help to close this gap. One simple thing we did for this project was to only start playing sounds once the user looks at the album art, which establishes a conceptual connection between the sound and the visuals.”

The team also had to contend with the technical limitations of a mobile chipset. 

“We need to balance variability in sonic experience with the very real need to keep our asset size low to improve effect performance, so in practice we have to be very economical with the number of audio files we include,” explains Nichols. “There’s so much great sonic material on this Haim album that we weren’t able to incorporate into the effect, but I think we did a good job of finding a representative collection of sounds that play well together and portray the overall vibe of this album in a compelling way.”

Embracing the ethereal

“The original vision for the audio content was to create a ‘dreamy journey through the new album,’ which, fortunately, fit with our technical restrictions,” adds Facebook Sound Designer and Composer Dren McDonald. “We knew that there could be issues related to time locking sounds or music loops, which would be inconsistent and device-dependent. This affected the types of sounds that I edited from the Haim tracks, so I avoided rhythmic/tempo-based sounds or musical phrases.”

Instead, McDonald focused on group and solo vocals as well as distinct musical sounds like guitars, bass lines, and the mandolin, which were designed to float over pedal tones created from various synth parts. “The pedal tones were the key ingredient to the concept, as we created three distinct pedal tones or root notes, which would change as the user tapped on different parts of the AR cover art,” he says. “When the pedal tone changed, it felt like a fresh start.”

Each pedal tone has its own set of specific sounds, for a truly unique experience. “In some cases I had to slightly adjust the pitch of the original sounds to get them to agree with the pedal tone,” says McDonald, “but I tried to set this up so that most of the sounds never had to be pitched more than one semi-tone, and that I never had to pitch the vocals (which would have come with some audio side effects that most of us would notice when the vocals were so exposed).”

The final step was to augment the sounds themselves.

“After choosing the sounds/phrases and pedal tones, I applied reverb and delay to all of the sounds (except the pedal tones), to add to the ‘dreamy’ quality that Creative Shop and the band were trying to achieve,” says McDonald. “This took a little bit of creative license on my part, and I wasn't sure how the band would react to my editing and manipulation of their music. Happily, this provided the vibe that everyone was looking for.” 

Music to our eARs

As AR grows steadily more prevalent in daily life and as we continue down the road to AR glasses, we’ll continue to see immersive experiences that delight people while giving them new outlets for creative expression. And music stands to play a prominent role.

“Music and AR are a match made in heaven,” says Moller. “With our new graphics and audio capabilities, we’re going to see visuals, music, and interaction continuing to feed into each other and evolve into new forms of music experience. The ability for those experiences to play out across multiple surfaces is going to open up a lot of opportunity for radical experimentation, and we’re going to see more and more compelling and norm-busting work.”

“Having worked alongside the Facebook Creative Shop team on several projects, they’re always really keen to push the envelope of what can be done in AR — and by virtue of that, the depth of my knowledge of what is possible has expanded,” Juniper adds. “It’s helped me to think of AR as a creative medium rather than just considering filters as another marketing asset. In the instance of Haim, we managed to make a really rich interactive experience that I hadn't thought possible before.”