Deepfakes have gone mainstream. Popular AI-generated videos of real people have already featured prominent government and business leaders, like former U.S. President Barack Obama and Facebook CEO Mark Zuckerberg. And some have led to great harm: an audio deepfake tricked the CEO of a European energy firm out of €220,000. He got a phone request from his chief executive – or so he thought. In fact, it was a deepfaked voice directing him to transfer the money.
Deepfakes are growing both more sophisticated and more accessible, and the implications are serious. But we also see ways that digital persona technology can be applied differently, for positive uses. And as deepfakes are more commonly used as a force for good, people will also grow more accustomed to spotting nefarious ones.
Deepfake technology has already been applied in multiple industries. Reuters developed an AI-based prototype where one of its real human anchors appears to read the news without the anchor himself recording the segment. In the retail space, deepfake-driven apps let people try out designer wear virtually and move around in incredibly realistic settings. And in health, generative adversarial networks – the technology that underpins deepfakes – create “fake” MRI scans showing brains with tumors. Why? The fake images help train algorithms to detect tumors in real patient MRIs.
Now, looking beyond the idea of “faking” living people, we’re using deepfake technology to explore the past to create a conversational digital persona. Imagine entering a museum and learning about an artist’s life and works not from signs on the wall or a printed brochure, but from the artist himself. What would be a passive museum experience becomes an interactive, immersive one, with famous creators, curators and personalities.
A vast combination of technologies makes this experience possible. Face superimposition and speech synthesis through deep learning create the synthetic video and audio. Natural Language Understanding and Natural Language Processing enable interactive dialogue. Text sentiment, voice sentiment and facial analysis provide the extra edge for the digital persona to be as responsive, expressive, immersive, proactive and adaptive as a real person would be.
At the heart of all of this is the idea of human centered computing. We work to foster engagement with the visitor, take human feedback into account to enhance the experience, and respect individual behavior during interactions. And most importantly – in stark contrast to malicious uses of deepfakes – we consider the ethical implications of using the technology.
Bringing this to life – so to speak! – requires more than just technology. We incorporate expertise from sociology, psychology and cognitive science. This lets us do much more than just create a visual representation of the artist reading a script. We can analyze user perception and attention span, while also improving ease of access to persons with disabilities. We can collect user feedback and improve the system’s long-term functionality. We can design the persona so that it can interact not only with individuals, but with larger groups, whether organized school groups touring a museum together, or simply a collection of friends.
Most important of all is embedding ethical design into the approach from the beginning. Accountability, transparency, ingenuity, fairness, and auditability are deeply ingrained in algorithmic decision-making in order to mitigate negative social impact. And in the case of a digital persona based on a real person – we need permissions and consent from the person or from the rightful heir in the his/her family. This is all in stark contrast to typical deepfakes, which are created to misinform. With the right approach, digital personas could soon emerge as a force for good! Stay tuned for updates on our efforts in this space.
To learn more on how digital personas can be applied for good, contact Sanjay Podder.
The authors would like to acknowledge the efforts of Nisha Ramachandra, Shalabh Kumar Singh, and Mitali Chatterjee in this work.