When we see a movie dubbed into Spanish, we usually suffer from various problems. On the one hand, there is hearing a different voice from famous actors, something that personally bothers me a lot, but the worst thing is seeing how the mouth makes a movement and the sound that comes out does not correspond to it.
Synchronizing the lips with the dubbed language is the objective of a new project that will use a technology similar to that used in deepfakes, technology that will digitally move the muscles of the face so that they have a better synchronization with what is being heard.
They will do it with a program called TrueSync, capable of analyzing the movement of the actor’s mouth and recording how it moves with each sound. This creates a digital double that will move the lips differently depending on the new sounds recorded.
The magic is in registering all the nuances of the actor’s performance, since it is important that everything is natural, that the face does not have a grimace that the actor would never make.
In a video posted by gizmodo we can see an example of how it was achieved with Tom Hanks speaking Japanese.
Although the idea is good, there are several problems that stand out:
– Not all actors want to see their face digitally altered so that it looks different in each country.
– The cost of dubbing, which is already expensive, will increase dramatically if a process of these characteristics must be carried out on each face of each folded film.
Be that as it may, it never ceases to amaze us how deepfakes are being used more and more on a day-to-day basis, and not just to cheat and play pranks.