Facial Expression Animation Research Articles

ExpCLIP: Bridging Text and Facial Expressions via Semantic Alignment

The objective of stylized speech-driven facial animation is to create animations that encapsulate specific emotional expressions. Existing methods often depend on pre-established emotional labels or facial expression templates, which may limit the necessary flexibility for accurately conveying user intent. In this research, we introduce a technique that enables the control of arbitrary styles by leveraging natural language as emotion prompts. This technique presents benefits in terms of both flexibility and user-friendliness. To realize this objective, we initially construct a Text-Expression Alignment Dataset (TEAD), wherein each facial expression is paired with several prompt-like descriptions. We propose an innovative automatic annotation method, supported by CahtGPT, to expedite the dataset construction, thereby eliminating the substantial expense of manual annotation. Following this, we utilize TEAD to train a CLIP-based model, termed ExpCLIP, which encodes text and facial expressions into semantically aligned style embeddings. The embeddings are subsequently integrated into the facial animation generator to yield expressive and controllable facial animations. Given the limited diversity of facial emotions in existing speech-driven facial animation training data, we further introduce an effective Expression Prompt Augmentation (EPA) mechanism to enable the animation generator to support unprecedented richness in style control. Comprehensive experiments illustrate that our method accomplishes expressive facial animation generation and offers enhanced flexibility in effectively conveying the desired style.

Learning a crowd-powered perceptual distance metric for facial blendshapes

It is known that purely geometric distance metrics cannot reflect the human perception of facial expressions. A novel perceptually based distance metric designed for 3D facial blendshape models is proposed in this paper. To develop this metric, comparative evaluations of facial expressions were collected from a crowdsourcing experiment. Then, the weights of a distance metric, based on descriptive features of the models, were optimized to match the results with crowdsourced data, through a metric learning process. The method incorporates perceptual properties such as curvature and visual saliency. A formal analysis of the results proves the high correlation between the metric output and human perception. The effectiveness and success of the proposed metric were also compared to other distance alternatives. The proposed metric will enable intelligent processing of 3D facial blendshapes data in several ways. It will be possible to generate perceptually valid clustering and visualization of 3D facial blendshapes. It will help reduce storage and computational requirements by removing redundant expressions that are perceptually identical from the overall dataset. It can also be used to assist novice animators while creating plausible and expressive facial animations.

Facial Expression Animation by Landmark Guided Residual Module

We study the problem of facial expression animation from a still image according to a driving video. This is a challenging task as expression motions are non-rigid and very subtle to be captured. Existing methods mostly fail to model these subtle expression motions, leading to the lack of details in their animation results. In this paper, we propose a novel facial expression animation method based on generative adversarial learning. To capture the subtle expression motions, Landmark guided Residual Module (LRM) is proposed to model detailed facial expression features. Specifically, residual learning is conducted at both coarse and fine levels conditioned on facial landmark heatmaps and landmark points respectively. Furthermore, we employ a consistency discriminator to ensure the temporal consistency of the generated video sequence. In addition, a novel metric named Emotion Consistency Metric is proposed to evaluate the consistency of facial expressions in the generated sequences with those in the driving videos. Experiments on MUG-Face, Oulu-CASIA and CAER datasets show that the proposed method can generate arbitrary expression motions on the source still image effectively, which are more photo-realistic and consistent with the driving video compared with results of state-of-the-art methods.

Processing Real-Life Recordings of Facial Expressions of Polish Sign Language Using Action Units

Automatic translation between the national language and sign language is a complex process similar to translation between two different foreign languages. A very important aspect is the precision of not only manual gestures but also facial expressions, which are extremely important in the overall context of a sentence. In this article, we present the problem of including facial expressions in the automation of Polish-to-Polish Sign Language (PJM) translation-this is part of an ongoing project related to a comprehensive solution allowing for the animation of manual gestures, body movements and facial expressions. Our approach explores the possibility of using action unit (AU) recognition in the automatic annotation of recordings, which in the subsequent steps will be used to train machine learning models. This paper aims to evaluate entropy in real-life translation recordings and analyze the data associated with the detected action units. Our approach has been subjected to evaluation by experts related to Polish Sign Language, and the results obtained allow for the development of further work related to automatic translation into Polish Sign Language.

Emotional 3D speech visualization from 2D audio visual data

Visual speech is hard to recreate by human hands because animation itself is a time-consuming task: both precision and detail must be considered and match the expectations of the developers, but above all, those of the audience. To solve this problem, some approaches has been designed to help accelerate the animation of characters faces, as procedural animation or speech-lip synchronization, where the most common areas for researching these methods are Computer Vision and Machine Learning. However, in general, these tools can have any of these main problems: difficulty on adapting to another language, subject or animation software, high hardware specifications, or the results can be receipted as robotic. Our work presents a Deep Learning model for automatic expressive facial animation using audio. We extract generic audio features from expressive audio speeches rich in phonemes for nonidiom focus speech processing and emotion recognition. From videos used for training, we extracted the landmarks for frame-speech targeting and have the model learn animation for phonemes pronunciation. We evaluated four variants of our model (two function losses and with emotion conditioning) by using a user perspective survey where the one using a Reconstruction Loss Function with emotion training conditioning got more natural results and score in synchronization with the approval of the majority of interviewees. For perception of naturalness, it obtained a 38.89% of the total votes of approval and for language synchronization obtained the highest average score with 65.55% (98.33 of a 150 total points) for English, German and Korean languages.

Animation of Bird Characters with Special Focus on Rio(2011)

This study analyzes the animation of bird characters based on real-world avian laws of motion in the 3D animated film, Rio (2011). The research focuses on three aspects—the primary movement, personification, and animation of facial expressions of birds. First, the primary movement of birds, including take-off and landing, are analyzed. Then, personification and animation of facial expressions are investigated based on some of the characters’ expressions in Rio (2011). Finally, different modes of animating avian characters’ expressions are discussed. This study is expected to contribute to future studies of similar topics in other films and lead to better understanding of the animation of avian characters.

Disentangling audio content and emotion with adaptive instance normalization for expressive facial animation synthesis

Abstract3D facial animation synthesis from audio has been a focus in recent years. However, most existing literature works are designed to map audio and visual content, providing limited knowledge regarding the relationship between emotion in audio and expressive facial animation. This work generates audio‐matching facial animations with the specified emotion label. In such a task, we argue that separating the content from audio is indispensable—the proposed model must learn to generate facial content from audio content while expressions from the specified emotion. We achieve it by an adaptive instance normalization module that isolates the content in the audio and combines the emotion embedding from the specified label. The joint content‐emotion embedding is then used to generate 3D facial vertices and texture maps. We compare our method with state‐of‐the‐art baselines, including the facial segmentation‐based and voice conversion‐based disentanglement approaches. We also conduct a user study to evaluate the performance of emotion conditioning. The results indicate that our proposed method outperforms the baselines in animation quality and expression categorization accuracy.

Transformasi Ruang 3D Pada Animasi Expresi Wajah Avatar Berbasis Radial Basis Function

One technique for forming facial animations on avatars is by reusing existing animations, either from other avatar animations or animations from motion data obtained using facial motion capture. This research focuses on the transformation of 3D space for the formation of facial animations on avatars in games or animated films. The transformation is carried out from motion capture data into a 3D avatar face model with 3 face models, namely the human face model, the swan face model and the anoman face model. The motion capture data is transferred according to the feature points of the face model. The results obtained by the facial model feature points will have animations that match the motion capture data. Of the 3 target face models used, the animation results with registration on the human face model have an average standard deviation is 0,0510. The goose face model has an average standard deviation is 0.0034 and the anoman face model has an average standard deviation is 0,0024. With this technique, it is hoped that the formation of facial expression animation on Avatar can be done more quickly because of the reuse of facial motion capture data.

애니메이션의 12원칙 중 역학에 관련된 원칙들과 실제 : 표정 애니메이팅을 중심으로

본 연구는 애니메이션 기본원칙 중 역학에 관련된 원칙들을 인간의 기본 감정표현인 행복, 슬픔, 분노, 혐오, 놀라움, 두려움 6가지 실제 표정 움직임과 비교하여 분석하고 애니메이터 관점에서 효과적인 표정애니메이팅에 적용 방법을 고찰한 것이다. 기존 표정 애니메이션 관련 논문과 연구들은 AI 및 머신러닝 적용 목적을 위한 또는 감정에 따른 얼굴 근육 움직임 분석 등이 대부분이고 애니메이터 관점에서 표정 애니메이팅에 있어서 필요한 얼굴 부위별 타이밍 및 움직임 시작 순서, 그리고 근육의 흐름을 분석은 부족했다. 이를 위해 애니메이션 기본원칙 중 역학에 관련된 원칙을 분류하고 부위별 얼굴 근육의 정의하며 근육의 흐름을 연구하였다. 연구한 내용을 인간의 기본 감정표현인 행복, 슬픔, 분노, 혐오, 놀라움, 두려움 6가지 표정을 중심으로 실제와 비교·분석하고 이를 애니메이터 관점에서 효과적 표정 애니메이팅을 위한 얼굴 부위별 움직임의 순서, 타이밍 등을 고찰하여 애니메이션 원칙의 보완점을 찾게 되었다. 표정은 감정에 의해 나오는 얼굴의 근육 움직임이기 때문에 대부분 근육의 움직임이 역학 관련 애니메이션 원칙을 따르지만 때로는 감정이 자극하는 부위에 따라 그 움직임의 순서가 바뀔 수도, 더 빠르게 또는 더 느리게 변할 수 있다는 것을 확인할 수 있었다.

Association of Game Events with Facial Animations of Computer-Controlled Virtual Characters Based on Probabilistic Human Reaction Modeling

Computer-controlled virtual characters are essential parts of most virtual environments and especially computer games. Interaction between these virtual agents and human players has a direct impact on the believability of and immersion in the application. The facial animations of these characters are a key part of these interactions. The player expects the elements of the virtual world to act in a similar manner to the real world. For example, in a board game, if the human player wins, he/she would expect the computer-controlled character to be sad. However, the reactions, more specifically, the facial expressions of virtual characters in most games are not linked with the game events. Instead, they have pre-programmed or random behaviors without any understanding of what is really happening in the game. In this paper, we propose a virtual character facial expression probabilistic decision model that will determine when various facial animations should be played. The model was developed by studying the facial expressions of human players while playing a computer videogame that was also developed as part of this research. The model is represented in the form of trees with 15 extracted game events as roots and 10 associated animations of facial expressions with their corresponding probability of occurrence. Results indicated that only 1 out of 15 game events had a probability of producing an unexpected facial expression. It was found that the “win, lose, tie” game events have more dominant associations with the facial expressions than the rest of game events, followed by “surprise” game events that occurred rarely, and finally, the “damage dealing” events.

Facial expression animation through action units transfer in latent space

Automatic animation synthesis has attracted much attention from the community. As most existing methods take a small number of discrete expressions rather than continuous expressions, their integrity and reality of the facial expressions is often compromised. In addition, the easy manipulation with simple inputs and unsupervised processing, although being important to the automatic facial expression animation applications, is relatively less concerned. To address these issues, we propose an unsupervised continuous automatic facial expression animation approach through action units (AU) transfer in the latent space of generative adversarial networks. The expression descriptor which is depicted with AU vector is transferred into the input image without the need of labeled pairs of images and even without their expressions and further network training. We also propose a new approach to quickly generate input image's latent code and cluster the boundaries of different AU attributes with their latent codes. Two latent code operators, vector addition and continuous interpolation, are leveraged for facial expression animation simulating align with the boundaries in the latent space. Experiments have shown that the proposed approach is effective on facial expression translation and animation synthesis.

3D face modeling from single image based on discrete shape space

AbstractIn this article, we propose a novel 3D face modeling method which constructs a new 3D face model from a low‐dimensional feature space consisted of a large set of blend shapes based on the discrete shape space theory. The details of original face features are completely retained during the modeling process and a large number of new natural faces are constructed by several face samples. The optimization process of our method is independently decoupled for different facial attributes (identity, expression, and head pose), which improves the application flexibility and reduces the probability of it falling into a local optimal situation. The new facial data with new attributes are constructed based on the geodesic path search in discrete shape space with sufficient freedom and accuracy. In experiments and applications based on public databases (Helen, LFW, and CUFS), the modeling results show our method can provide high‐quality 3D face model, with enough freedom for face expression editing and natural facial expression animation from a small facial sample set.

Research on 3D Facial Expression Simulation Technology

In order to generate realistic facial animation, this paper uses depth learning object detection technology to detect the three wrinkles of human face, and then the adaptive threshold algorithm is used to extract the wrinkle curves, finally applies realistic wrinkle animation to the virtual face model. In the aspect of expression animation simulation, based on MPEG-4 standard, a database of expression parameters covering six major expressions has been established, which divide the human face into four main animation units and other auxiliary animation units. According to the displacement of animation unit between neutral expression and other expressions, the facial bone points of virtual model are driven to move, thus realizing realistic facial expression animation.

Face Dynamic Modeling Based on Deep Learning and Feature Extraction

Facial modeling is a key step to model visual effects in special movie effects and computer games. In this paper, a method based on the combination of deep learning and feature extraction is proposed for the modeling of 3D face model. Firstly, the face region is located for the captured face image. And then, the facial feature points are extracted by the landmark algorithm and the Convolutional Neural Network (CNN) is used to classify the facial expressions. Next, a special expression 3D face model is created by the deformation of the standard 3D face model based on the facial expressions classification result. Finally, the 3D face model and the extracted facial feature points are combined to perform personalized adjustment of the 3D model to complete a 3D facial expression animation system. The experimental results show that the proposed method can effectively perform the dynamic 3D face modeling which has high reality.

The multimodal interactional organization of tasting: Practices of tasting cheese in gourmet shops

Taste is a central sense for humans and animals, and it has been largely studied either from physiological and neurological approaches or from socio-cultural ones. This paper adopts another view, focused on the activity of tasting rather than on the sense of taste, approached within the perspective of ethnomethodology and multimodal conversation analysis. This view addresses the activity of tasting as it is interactionally organized in specific social settings, observed in a naturalistic way, on the basis of video recordings. Focusing on video recorded improvised tastings of cheese in gourmet shop encounters, the paper offers a systematic analysis of the way in which tasting is orderly achieved in an intersubjective way. It follows the various steps characterizing tasting, from the invitation to taste, to the grasping of a bit to taste, which is put in the mouth, chewed, and swallowed; it details how an interactional moment offering the taster a priviledged, individual, focused space in which to devote exclusive attention to the object tasted is actively tailored by all parties. By contrast, the completion of tasting is marked by a return to mutual gaze, the animation of facial expressions and nods, and the final production of a judgment of taste. By offering a systematic reconstruction of how these tasting moments are organized, the paper invites to a multimodal approach of sensoriality in social interaction.

Emotional facial expression transfer from a single image via generative adversarial nets

AbstractFacial expression transfer from a single image is a challenging task and has drawn sustained attention in the fields of computer vision and computer graphics. Recently, generative adversarial nets (GANs) have provided a new approach to facial expression transfer from a single image toward target facial expressions. However, it is still difficult to obtain a sequence of smoothly changed facial expressions. We present a novel GAN‐based method for generating emotional facial expression animations given a single image and several facial landmarks for the in‐between stages. In particular, landmarks of other subjects are incorporated into a GAN model to control the generated facial expression from a latent space. With the trained model, high‐quality face images and a smoothly changed facial expression sequence can be effectively obtained, which are showed qualitatively and quantitatively in our experiments on the Multi‐PIE and CK+ data sets.

3D visual simulation of individual and crowd behavior in earthquake evacuation

Simulation of behaviors in emergencies is an interesting subject that helps to understand evacuation processes and to give out warnings for contingency plans. Individual and crowd behaviors in the earthquake are different from those under normal circumstances. Panic will spread in the crowd and cause chaos. Without considering emotion, most existing behavioral simulation methods analyze the movement of people from the point of view of mechanics. After summarizing existing studies, a new simulation method is discussed in this paper. First, 3D virtual scenes are constructed with the proposed platform. Second, an individual cognitive architecture, which integrates perception, motivation, behavior, emotion, and personality, is proposed. Typical behaviors are analyzed and individual evacuation animations are realized with data captured by motion capture devices. Quantitative descriptions are presented to describe emotional changes in individual evacuation. Facial expression animation is used to represent individuals’ emotions. Finally, a crowd behavior model is designed on the basis of a social force model. Experiments are carried out to validate the proposed method. Results showed that individuals’ behavior, emotional changes, and crowd aggregation can be well simulated. Users can learn evacuation processes from many angles. The method can be an intuitional approach to safety education and crowd management.

Research On 3D Emotional Face Animation Based on Dirichlet Free Deformation Algorithm

In the process of people's daily communication, the human face has played a decisive role, it includes joy, anger, sadness, music and other complex expressions, and also bear the function of language expression. Therefore, the construction of three-dimensional facial expression animation model and synthesis technology in computer graphics is one of the hot spots. How do you create realistic facial expressions and actions on virtual 3D people? This paper introduces the 3D facial expression animation system based on the Dirichlet free deformation algorithm. The system consists of two parts: the first part, using the Natural Point facial expression capture system to collect facial expression data; the second part, the use of Autodesk Maya animation software to establish the corresponding Face 3D model, and then use the Dirichlet free deformation algorithm as a deformation technique. Experimental results show that the system can be very good and real-time simulation of human (hi, anger, sad, music, etc.) complex expression. This study is of great help to human-computer interaction systems.

Design of unsupervised facial expression animation based on geometric grid measurement

Many actual application images in the real world are formed by high dimensional data in most cases, while the manifold learning algorithm can explore the nonlinear information hidden in these high dimensional data. As most of manifold learning algorithms can only be defined in training cluster, it is impossible to project the sample on the lower dimensional space. In the thesis, we introduce a kind of double manifold algorithm based on LLE and Isomap. Different from the traditional LLE algorithm, our algorithm learns two kinds of manifold information in which one group of data relates to many types and it compares two kinds of single LLE algorithm and Isomap algorithm through the setting of the appropriate nearest neighbour number K. No matter for the recognition rate or running time, it is obviously superior to the other two kinds of algorithms and it can effectively achieve the estimation of facial expression and significantly reduce the computation complexity.

Design of unsupervised facial expression animation based on geometric grid measurement

Many actual application images in the real world are formed by high dimensional data in most cases, while the manifold learning algorithm can explore the nonlinear information hidden in these high dimensional data. As most of manifold learning algorithms can only be defined in training cluster, it is impossible to project the sample on the lower dimensional space. In the thesis, we introduce a kind of double manifold algorithm based on LLE and Isomap. Different from the traditional LLE algorithm, our algorithm learns two kinds of manifold information in which one group of data relates to many types and it compares two kinds of single LLE algorithm and Isomap algorithm through the setting of the appropriate nearest neighbour number K. No matter for the recognition rate or running time, it is obviously superior to the other two kinds of algorithms and it can effectively achieve the estimation of facial expression and significantly reduce the computation complexity.