High-fidelity, Directable Animation Transfer using Facial Decomposition on Optimized Micro-sequences (1 postdoctoral fellow)

We are looking for postdoctoral fellow candidates knowledgeable in computer graphics, machine learning, as well as computer vision. As the candidate should begin as soon as possible, we will prioritize candidates who can immediately work in Canada (Canadians, hold permanent residency, or already have a work permit for Canada).

The entertainment industry delivers media products around the world to people speaking a large number of distinct languages. Media products such as TV or internet spots, movies and video games, typically consist of a video and sound stream. To accommodate different languages, the producers typically re-generate the sound stream in different language in a post-production step, a process known as dubbing. While it still takes a lot of effort to re-generate the sound stream (dubbing), one of the remaining open challenges both from a research and production point of view is how to synchronize the original video stream with the new audio stream. The lack of synchronization creates a very uncanny effect that takes away from the experience since the lip motion of the actors does not correspond to the soundtrack. In this project we propose a practical solution to the dubbing synchronization problem, producing natural results that can be used in a production studio.

To achieve that, we created a strategic partnership with a dubbing studio with whom we will collaborate on this project. We propose a video synthesis method that blends animation focusing on the lip area while preserving the facial expression of the original actor. Our method will use novel optimization techniques to avoid the common undesired uncanny valley effect of video synthesis. Here is a good example of work we will try to outperform: http://gvv.mpi-inf.mpg.de/projects/VisualDubbing/ Montréal is quite bilingual and someone who knows English can do all of their day-to-day activities without any problem. As the candidate should begin as soon as possible, we will prioritize candidates who can immediately work in Canada (Canadians, hold permanent residency, or already have a work permit for Canada). We are looking for postdoctoral fellow candidates knowledgeable in computer graphics, machine learning, as well as computer vision.

Connaissances requises

The ideal postdoctoral fellow candidate has Master’s and PhD degrees with topics from at least two topics from computer graphics, machine learning, image processing, and computer vision. The postdoctoral fellow candidate studied to some extent many of the following topics: facial animation, character animation, geometric processing, video processing, image processing, computer vision, machine learning, and neural networks.
The candidates should be able C++ programmer. Experience with PyTorch, TensorFlow, OpenCV, Matlab, Cuda is an asset.

Programme d'études visé

Postdoctorat

Domaines de recherche

Technologies de l'information et des communications

Financement

Competitive funding is available for this project.

Autres informations

Date de début : 2019-07-01 

Partenaires impliqués : Concordia University, AudioZ

Personne à contacter

Eric Paquette | eric.paquette@etsmtl.ca