Expression-tailored talking face generation with adaptive cross-modal weighting

Abstract

The key of talking face generation is to synthesize the identity-preserving natural facial expressions with accurate audio-lip synchronization. To accomplish this, it requires to disentangle and fuse the latent features from multiple modalities, including the visual identity, facial expressions, and audio, etc. In this paper, we propose an end-to-end Expression-Tailored Generative Adversarial Network with Adaptive Cross-modal Weighting (ET-GAN-ACW). Different from previous talking face generation based on the identity image and audio, an expression video of arbitrary identity serves as the source in our system. On the one hand, multiple encoders are presented to disentangle the expression-tailored representation, audio-lip embedding, and face position localization in parallel. Additionally, instead of using a single image as the target identity, a multi-image identity encoder is proposed by exploring the different views of faces and merging them into a unified representation. These informative features from different modalities are then adaptively weighted and fused by the proposed Adaptive Cross-modal Weighting (ACW) mechanism. On the other hand, multiple discriminators are exploited to create the image-aware and video-aware realistic details, including a frame discriminator for the frame authenticity, and a spatial–temporal discriminator for the visual coherence of facial expression movements. Extensive quantitative evaluations on reconstruction error, identity preserving, expression retention, and audio-visual synchronization verify the superiority of our method. Qualitative results also demonstrate the effectiveness of our method in generating high-quality talking face videos.