High-fidelity, Directable Animation Transfer using Facial Decomposition on Optimized Micro-sequences

We are looking for postdoctoral fellow candidates knowledgeable in computer graphics, machine learning, as well as computer vision.

The entertainment industry delivers media products around the world to people speaking a large number of distinct languages. Media products such as TV or internet spots, movies and video games, typically consist of a video and sound stream.

To accommodate different languages, the producers typically re-generate the sound stream in different language in a post-production step, a process known as dubbing. While it still takes a lot of effort to re-generate the sound stream (dubbing), one of the remaining open challenges both from a research and production point of view is how to synchronize the original video stream with the new audio stream.

The lack of synchronization creates a very uncanny effect that takes away from the experience since the lip motion of the actors does not correspond to the soundtrack. In this project we propose a practical solution to the dubbing synchronization problem, producing natural results that can be used in a production studio.

To achieve that, we created a strategic partnership with a dubbing studio with whom we will collaborate on this project. We propose a video synthesis method that blends animation focusing on the lip area while preserving the facial expression of the original actor. Our method will use novel optimization techniques to avoid the common undesired uncanny valley effect of video synthesis. Here is a good example of work we will try to outperform: visual dubbing

The ideal time frame would be for the postdoctoral fellow to start in September 2019. Montréal is quite bilingual and someone who knows English and very basic French can do all of their day-to-day activities without any problem.   
 

Required knowledge

We are looking for postdoctoral fellow candidates knowledgeable in computer graphics, machine learning, as well as computer vision.

The ideal postdoctoral fellow candidate has Master’s and PhD degrees with topics from at least two topics from computer graphics, machine learning, image processing, and computer vision.

The postdoctoral fellow candidate studied to some extent many of the following topics:

  • facial animation
  • character animation
  • geometric processing
  • video processing
  • image processing
  • computer vision
  • machine learning
  • neural networks.

The candidates should be capable C++ programmer. Experience with OpenCV, Matlab, OpenGL, Direct3D, Cuda, OpenMP, OpenMPI, Boost, eigen, Unity, Houdini, or Blender is an asset.

Desired program of studies

Postdoctoral studies

Research domains

Information and communications technologies

Financing

Competitive funding is available for this project

Additional information

Date de début : 2019-09-01