Seeing sound: The new role of extended reality in music
Each year the Technology Vision team hosts events around the world to celebrate the launch of our new trend report. I found this year’s San Francisco event particularly special: entertainment was provided by a live band accompanied by Shimon, the marimba playing robot. Shimon uses artificial intelligence to listen and respond to pieces of music it hears, so that it can “jam” with human musicians in real time. While this robot’s presence might have been most interesting to those in the audience with backgrounds in AI and robotics, I enjoyed the experience of listening to Shimon for a different reason.
When I watched Shimon play, what jumped out to me was the connection between music and Extended Reality, a trend in the 2018 Accenture Technology Vision. The Extended Reality (XR) trend focuses on the ways experiences are being reshaped by virtual, augmented, and mixed reality technologies. One of the things about XR that excites me the most is that it represents a nascent medium for creative expression. While using XR technologies today still involves looking at objects on or through a screen, this information is overlaid on top of a digital or real-world environment. The viewer gets access to a sense of space not found using a phone, computer, or TV. It’s not just looking at something; it’s experiencing something.
Listening to music in extended reality environments adds depth to the discussion around the so-called “Walkman Effect,” which is the idea that listening to music with headphones adds a new layer of experience to someone’s everyday activities.
 When the Walkman was first introduced, it brought a lot of attention to the idea that people could now use music and headphones to reshape reality – to create a version of reality that they alone experienced. XR raises the idea of the Walkman Effect to entirely new heights.
Companies are already experimenting with the opportunities this creates, collaborating with musicians to develop auditory experiences in new, technology-enhanced environments. The Icelandic band Sigur Rós has created an entirely new listening experience with Magic Leap, a company developing augmented reality goggles. This new experience lets individuals move around any given room, literally walking into and discovering new pieces of the band’s composition. Björk, a singer famous for experimentation in musical style over her four-decade career, is also among a growing number of musicians experimenting with XR. She’s worked with VR technologies to showcase her 2015 album Vulnicura. Several of the album’s songs have been given VR “music videos,” which are experienced spatially in ways unlike a typical music video. The artist has even performed songs wearing costumes that can be overlaid with AR effects, giving concert-goers another view of a song’s performance.
In each of these examples, the focus is on using extended reality technologies to “add” to the standard experience of listening to music. But other companies are working on a related challenge: making virtual experiences seem more real by recreating the natural qualities of sound. As extended reality becomes more sophisticated and commonplace, anything that makes the experience feel “fake” – like sound that doesn’t match what we would expect in reality – will hinder XR’s adoption.
Leaders in the space are working to help XR developers create “virtual” auditory experiences that mimic the way sound waves interact with a physical space. In other words, when you hear a sound inside a “room” that only exists in virtual reality goggles, that sound should echo, reverberate, fade, or otherwise behave just as it would if you were standing in a physical version of that same room. Google’s Resonance Audio SDK is one example of a framework VR creators can use to realistically model 3D sounds without compromising application speed – addressing the need to make VR “feel real” in real time.
As XR technologies become commonplace, businesses will need to deliver new experiences without compromising on key elements like sound, which underpin the “real” experiences on which XR looks to build. Businesses who are attentive to the balance between delivering an exciting virtual experience, and delivering one that is familiar enough to feel “real,” will lay the foundation that makes XR technologies part of our shared everyday realities.
For a larger discussion about the revolutionary potential of XR technologies, be sure to read this year’s Technology Vision trend “Extended Reality: The End of Distance.”.
 Hosokawa, S. (1984). The Walkman Effect. Popular Music, 4, 165–180.