Realistic 3D Facial Animation Research Articles

Expressive 3D Facial Animation Generation Based on Local-to-Global Latent Diffusion.

3D Facial animations, crucial to augmented and mixed reality digital media, have evolved from mere aesthetic elements to potent storytelling media. Despite considerable progress in facial animation of neutral emotions, existing methods still struggle to capture the authenticity of emotions. This paper introduces a novel approach to capture fine facial expressions and generate facial animations using audio synchronization. Our method consists of two key components: First, the Local-to-global Latent Diffusion Model (LG-LDM) tailored for authentic facial expressions, which can integrate audio, time step, facial expressions, and other conditions towards possible encoding of emotionally rich yet latent features in response to possibly noisy raw audio signals. The core of LG-LDM is our carefully designed Facial Denoiser Model (FDM) for aligning the local-to-global animation feature with audio. Second, we redesign an Emotion-centric Vector Quantized-Variational AutoEncoder framework (EVQ-VAE) to finely decode the subtle differences under different emotions and reconstruct the final 3D facial geometry. Our work significantly contributes to the key challenges of emotionally realistic 3D facial animation for audio synchronization and enhances the immersive experience and emotional depth in augmented and mixed reality applications. We provide a reproducibility kit including our code, dataset, and detailed instructions for running the experiments. This kit is available at https://github.com/wangxuanx/Face-Diffusion-Model.

Svara Rachana - Audio Driven Facial Expression Synthesis

Abstract: Svara Rachana is a fusion of artificial intelligence and facial animation which aims to revolutionize the field of digital communication. Harnessing the ever-evolving power of neural networks in the form of Long Short-Term Memory (LSTM) model, Svara Rachana offers a cutting edge, interactive web application designed to synchronize human speech with realistic 3D facial animation. Users can upload or record an audio file and upload it to the web interface containing human speech, with the core functionality being the generation of synchronized lip movements on a 3D avatar. The system gives special emphasis on the accuracy of the system to generate reliable facial animation movements. By providing an interactive, human like 3D model, Svara Rachana aims to make machine to human interaction a more impactful experience by blurring the lines between humans and machines.

Geometry-Guided Dense Perspective Network for Speech-Driven Facial Animation.

Realistic speech-driven 3D facial animation is a challenging problem due to the complex relationship between speech and face. In this paper, we propose a deep architecture, called Geometry-guided Dense Perspective Network (GDPnet), to achieve speaker-independent realistic 3D facial animation. The encoder is designed with dense connections to strengthen feature propagation and encourage the re-use of audio features, and the decoder is integrated with an attention mechanism to adaptively recalibrate point-wise feature responses by explicitly modeling interdependencies between different neuron units. We also introduce a non-linear face reconstruction representation as a guidance of latent space to obtain more accurate deformation, which helps solve the geometry-related deformation and is good for generalization across subjects. Huber and HSIC (Hilbert-Schmidt Independence Criterion) constraints are adopted to promote the robustness of our model and to better exploit the non-linear and high-order correlations. Experimental results on the public dataset and real scanned dataset validate the superiority of our proposed GDPnet compared with state-of-the-art model. The code is available for research purposes at http://cic.tju.edu.cn/faculty/likun/projects/GDPnet.

Lip Viseme Analysis of Chinese Shaanxi Xi’an Dialect Visual Speech for Talking Head in Speech Assistant System

Recently, multimedia, especially animation, has played an important role in language learning. It has made a significant contribution to the language learning process of the hearing impaired for learners of all ages, especially the 3D animation of virtual teachers in computer-assisted language learning applications. Researches on viseme in various official languages and 3D facial animations have emerged in various countries. This paper has done lots of work in order to created Chinese Shaanxi Xi’an Dialect viseme data base by lip viseme analysis. Series of lip visemes frame data was gained by processing this dialect images and videos taken by camera. The classification of the lip viseme of this dialect also made based on the lip viseme classification of Mandarin. This work has great significance to create realistic 3D facial animation of this dialect.

Creative tools for producing realistic 3D facial expressions and animation

Creative exploration of realistic 3D facial animation is a popular but very challenging task due to the high level knowledge and skills required. This forms a barrier for creative individuals who have limited technical skills but wish to explore their creativity in this area. This paper proposes a new technique that facilitates users' creative exploration by hiding the technical complexities of producing facial expressions and animation. The proposed technique draws on research from psychology, anatomy and employs Autodesk Maya as a use case by developing a creative tool, which extends Maya's Blend Shape Editor. User testing revealed that novice users in the creative media, employing the proposed tool can produce rich and realistic facial expressions that portray new interesting emotions. It reduced production time by 25% when compared to Maya and by 40% when compared to 3DS Max equivalent tools.