Large motion anime head animation using a cascade pose transform network

Abstract

We study the problem of talking head animation from a single image where a target anime talking head is generated to mimic the change of facial expression and head movement of source anime characters. Most existing methods focus on generating talking heads from real humans. However, few efforts have been made to create anime talking head. Compared with human head generation, the key challenges of anime head generation are: how to align the pose and facial expression of the target head with that of the source head without explicit facial landmarks. To address this, we propose CPTNetV2, a cascaded pose transform network that unifies face pose transformation and head pose transformation. At the core of CPTNetV2 is the implicit encoding of facial changes and head movement by a pose vector. Given the pose vector, we introduce a mask generator to animate facial expression (e.g., close eyes and open mouth) and a grid generator to simulate head movement, followed by a fusion module to generate talking heads. To tackle large displacement and improve the quality of generation, we further design a details inpainting module with pose vector decomposition to reduce the receptive field of network required for pose transformation. In particular, we collect an anime talking head dataset AniHead-2K that includes around 2000 anime characters with different face/head poses. Extensive experiments on AniHead-2K demonstrate that CPTNetV2 can achieve arbitrary pose transformation conditioned on the target pose vector and outperforms other state-of-the-art methods. We also verify the effectiveness of each module through ablative studies. Additional results show that CPTNetV2 has good generalization and is applicable to generate anime talking head even based on human videos. The dataset will be made available at: https://github.com/zhangjiale487/AniHead-2K.