UCL computer scientists create photorealistic, talking digital actors

The 3D Vision team at UCL developed a new method to synthesise video of photorealistic human faces in speech, resulting in the creation of a highly successful spinout company Synthesia.

The technology behind Synthesia, developed by Professor Lourdes Agapito (UCL Department of Computer Science) and team, enables users to create professional looking videos with photorealistic digital ‘actors’ simply typing a message. The results are indistinguishable from real video but without the need for cameras, actors, or expensive film studios.

Synthesia’s technology was used to make David Beckham speak nine languages as part of a 2019 global campaign video to raise awareness about malaria, resulting in 700 million online impressions.

A cheaper and easier method for 3D capture  

Synthesising photorealistic, expressive human faces in speech has been a long-standing challenge in computer vision and graphics. For decades, this technology has been the exclusive domain of the film and TV industries, with multi-million budgets needed to build specialised and complex multi-camera 3D capture studios to create digital 3D doubles of humans, and for manual post-production by visual effects artists.  

Professor Agapito’s team pioneered weakly supervised methods for 3D human pose estimation from single images that only require 2D image annotations. These are cheaper and easier to harvest than the 3D annotations required by other methods. These breakthrough algorithms for monocular non-rigid 3D reconstruction by Professor Agapito’s team at UCL form the underpinning technology that made 3D-driven, photorealistic and low-cost AI video synthesis finally possible and form an integral part of Synthesia’s technology.  

Scoping a range of commercial applications

Professor Agapito co-founded Synthesia with other researchers and entrepreneurs to provide commercial solutions for a range of applications for this new technology, from lip-sync dubbing for content localisation to personalised video messages and corporate training. Synthesia technology allows users to create professional looking videos by simply typing a message, using an automated, 3D-driven AI process to synthesize photorealistic results that are indistinguishable from real video but without the need for cameras, actors, or expensive film studios.

Synthesia has transitioned from offering high profile video-to-video services towards offering a ‘Software as a Service’ platform where users can create videos simply by writing the speech of the digital ‘actor’. This technology has a wide range of applications, from corporate training to in-house communication to sales. As such, Synthesia’s services have been used by diverse clients, including Reuters, WPP, Dixa, Just Eat, Tesco, FedEx, Facebook and Google. As a result of its world-leading technology, Forbes magazine named Synthesia one of its ‘fearless five’ Tech companies.

Public health engagement

Synthesia’s technology was also crucial in a 2019 Malaria No More campaign that raised $14 billion to help end the world’s three biggest preventable killer diseases: AIDS, Tuberculosis and malaria. It was used to make David Beckham speak nine languages as part of a 2019 campaign video, and because this video localised the campaign to suit specific global audiences, it created 700 million online impressions and resulted in the disease’s peak awareness in almost three years.

Synthesia’s CEO and co-founder Victor Riparbelli commented:

“The ability to capture the 3D geometry and appearance of a human face in speech from a single video with algorithms building on Agapito’s research has been transformational in allowing us to build a low-cost solution to create high fidelity 3D avatars of humans for animation and synthesis.”

Links

• Professor Lourdes Agapito’s academic profile
• UCL Computer Science
• UCL Faculty of Engineering Sciences
• UCL Engineering REF 2021

Image 

• Image credit: iStock / Maksim Tkachenko