Facial Expression Generation Research Articles

A highly naturalistic facial expression generation method with embedded vein features based on diffusion model

Abstract Facial expression generation technology has achieved notable progress in computer vision and artificial intelligence. However, challenges persist regarding background consistency, expression clarity, and detailed representation. Additionally, the instability of generative adversarial networks (GANs) during training affects both image quality and diversity. While diffusion models have demonstrated potential advantages over GANs, research on controllable expression generation remains limited. To address these challenges, this paper proposes a highly natural facial expression generation method based on denoising diffusion implicit models (DDIM) with embedded vein features. This approach avoids adversarial training by employing gradual diffusion to generate specific expressions, thereby enhancing both the diversity and authenticity of the images. Vein features are introduced and embedded within the generated expression images to protect the intellectual property of algorithm-generated digital resources. Firstly, image and expression text guide words are combined as conditional inputs to improve the authenticity and diversity of the generated images. Secondly, a classification coding network is introduced to guide expression generation, thus enhancing the accuracy and consistency of the produced expressions. Furthermore, this paper proposes a vein feature fusion method based on multi-directional local dynamic feature coding operator (MLDFO) and integrates DDIM with frequency-domain watermarking technology to achieve image intellectual property protection. Experimental results demonstrate the effectiveness of this method across several public datasets, including FFHQ, CelebA, FV-USM, and SDUMLA-HMT. Notably, in the CelebA dataset, the average expression recognition rate increased by 11.41%, with a 100.00% recognition rate for happy expressions. The generated expression images exhibit a high degree of authenticity and consistency, and the video conversion tests reveal a natural and smooth effect. These results confirm that this method not only advances facial expression generation technology but also significantly enhances the steganographic protection of images.

Empowering sign language communication: Integrating sentiment and semantics for facial expression synthesis

Translating written sentences from oral languages to a sequence of manual and non-manual gestures plays a crucial role in building a more inclusive society for deaf and hard-of-hearing people. Facial expressions (non-manual), in particular, are responsible for encoding the grammar of the sentence to be spoken, applying punctuation, pronouns, or emphasizing signs. These non-manual gestures are closely related to the semantics of the sentence being spoken and also to the utterance of the speaker’s emotions. However, most Sign Language Production (SLP) approaches are centered on synthesizing manual gestures and do not focus on modeling the speaker’s expression. This paper introduces a new method focused in synthesizing facial expressions for sign language. Our goal is to improve sign language production by integrating sentiment information in facial expression generation. The approach leverages a sentence’s sentiment and semantic features to sample from a meaningful representation space, integrating the bias of the non-manual components into the sign language production process. To evaluate our method, we extend the Fréchet gesture distance (FGD) and propose a new metric called Fréchet Expression Distance (FED) and apply an extensive set of metrics to assess the quality of specific regions of the face. The experimental results showed that our method achieved state of the art, being superior to the competitors on How2Sign and PHOENIX14T datasets. Moreover, our architecture is based on a carefully designed graph pyramid that makes it simpler, easier to train, and capable of leveraging emotions to produce facial expressions. Our code and pretrained models will be available at: https://github.com/verlab/empowering-sign-language.

Key‐point‐guided adaptive convolution and instance normalization for continuous transitive face reenactment of any person

AbstractFace reenactment technology is widely applied in various applications. However, the reconstruction effects of existing methods are often not quite realistic enough. Thus, this paper proposes a progressive face reenactment method. First, to make full use of the key information, we propose adaptive convolution and instance normalization to encode the key information into all learnable parameters in the network, including the weights of the convolution kernels and the means and variances in the normalization layer. Second, we present continuous transitive facial expression generation according to all the weights of the network generated by the key points, resulting in the continuous change of the image generated by the network. Third, in contrast to classical convolution, we apply the combination of depth‐ and point‐wise convolutions, which can greatly reduce the number of weights and improve the efficiency of training. Finally, we extend the proposed face reenactment method to the face editing application. Comprehensive experiments demonstrate the effectiveness of the proposed method, which can generate a clearer and more realistic face from any person and is more generic and applicable than other methods.

4D Facial Expression Diffusion Model

Facial expression generation is one of the most challenging and long-sought aspects of character animation, with many interesting applications. The challenging task, traditionally having relied heavily on digital craftspersons, remains yet to be explored. In this paper, we introduce a generative framework for generating 3D facial expression sequences (i.e. 4D faces) that can be conditioned on different inputs to animate an arbitrary 3D face mesh. It is composed of two tasks: (1) Learning the generative model that is trained over a set of 3D landmark sequences, and (2) Generating 3D mesh sequences of an input facial mesh driven by the generated landmark sequences. The generative model is based on a Denoising Diffusion Probabilistic Model (DDPM), which has achieved remarkable success in generative tasks of other domains. While it can be trained unconditionally, its reverse process can still be conditioned by various condition signals. This allows us to efficiently develop several downstream tasks involving various conditional generation, by using expression labels, text, partial sequences, or simply a facial geometry. To obtain the full mesh deformation, we then develop a landmark-guided encoder-decoder to apply the geometrical deformation embedded in landmarks on a given facial mesh. Experiments show that our model has learned to generate realistic, quality expressions solely from the dataset of relatively small size, improving over the state-of-the-art methods. Videos and qualitative comparisons with other methods can be found at https://github.com/ZOUKaifeng/4DFM. Code and models will be made available upon acceptance.

인공지능 기반 심리상담에 활용되는 딥러닝 기법 고찰

Objectives This study aims to review and summarize deep learning techniques that have been developed to date and to discuss how they can be applied in practice to artificial intelligence-based psychology counseling robots.
 Methods To examine the latest deep learning techniques utilized in artificial intelligence psychological counseling, an extensive literature search was conducted for research papers and relevant books from both domestic and in-ternational sources in the fields of emotion analysis, facial expression generation, and text generation.. For do-mestic literature, databases such as RISS and KISS were utilized, and for international literature, research paper search sites including ERIC (ProQuest), JSTOR, and Google Scholar were employed.
 Results When applying artificial intelligence in psychological counseling, the prominent technology of interest in-volves recognizing emotions from the interlocutor's facial expressions and generating similar expressions, or rec-ognizing emotions from sentences spoken by the interlocutor and generating responsive sentences. The study in-troduced the concepts of machine learning and deep learning and explained the workings of deep learning techni-ques, such as CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), LSTM (Long Short Term Memory), Transformer, BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pretrained Transformer), and GAN (Generative Adversarial Network). These techniques are founded on the princi-ples derived from human brain functioning. Particularly, emotion recognition programs through image analysis have predominantly relied on CNN, while research in the realm of language has advanced with the utilization of GPT-2, LSTM, and similar methods.
 Conclusions Among the existing deep learning techniques, the recognition of emotions from the interlocutor's fa-cial expressions and the generation of similar expressions, or the recognition of emotions from sentences spoken by the interlocutor and the generation of responsive sentences, are the technologies predominantly under scrutiny for their application in artificial intelligence-based psychological counseling. Nevertheless, rule-based chatbots continue to dominate the landscape. This paper explored the merits and demerits of AI-based counseling em-ploying such deep learning techniques. Looking ahead, there is a need for the advancement of technology that en-ables natural interactions, coupled with the essential verification of its effectiveness.

Contour wavelet diffusion – a fast and high-quality facial expression generation model

Facial expressions are important for conveying information in human interactions. The diffusion model can generate high-quality images for clearer and more discriminative faces, but its training and inference time is often prolonged, hampering practical application. Latent space diffusion models have shown promise in speeding up training by leveraging feature space parameters, but they require additional network structures. To address these limitations, we propose a contour wavelet diffusion model that accelerates both training and inference speeds. We use a contour wavelet transform to extract components from images and features, achieving substantial acceleration while preserving reconstruction quality. A normalised random channel attention module enhances the quality of generated images by focusing on high-frequency information. We also include a reconstruction loss function to enhance convergence speed. Experimental results demonstrate the effectiveness of our approach in boosting the training and inference speeds of diffusion models without sacrificing image quality. Fast generation of facial expressions can provide a smoother and more natural user experience, which is important for real-time applications. In addition, the increase in inference speed can save the use of computational resources, reduce system cost and improve energy efficiency, which is conducive to promoting the development and application of this technology.

4D facial analysis: A survey of datasets, algorithms and applications

Facial information plays an important role in human communication, e.g., rich and nuanced facial expressions effectively convey emotions. Traditional data such as 2D facial images and videos are susceptible to perturbation from lighting and occlusion, while 3D static data such as mesh models lack the temporal information which is necessary to describe facial dynamics. Therefore, researches on 4D facial data (3D facial models together with time as the fourth dimension) have received considerable attention in recent years. 4D data can simultaneously reflect the complex facial temporal and space information. To fully explore these characteristics, we present a systematic overview of the 4D facial research in this paper. We give a review in terms of historical development of 4D facial datasets, acquisition process of these datasets, related algorithms and applications, and discussion of outstanding issues. We also analyze the 4D facial research works by summarizing and comparing them; in particular, we present the results on three kinds of tasks conducted on the 4D facial datasets, i.e., expression-related tasks (AU detection, expression recognition and retrieval), generation tasks (3D facial reconstruction, facial expression and facial animation generation), and other tasks (facial registration, facial recognition and facial disease diagnosis). Finally, we summarize the future research directions.

MODEL OF MIMIC EXPRESSIONS OF HUMAN EMOTIONAL STATES FOR THE VIDEO SURVEILLANCE SYSTEMS

The presented paper proposes a novel computational model for generating facial expressions that mimic human emotional states. The authors aim to create a system that can generate realistic facial expressions to be used in human-robot interactions. The proposed model is based on the Facial Action Coding System, a widely used tool for describing facial expressions. FACS is used in this study to identify the muscles involved in each facial expression and the degree to which each muscle is activated. Several machine-learning techniques were utilized to learn the relationships between facial muscle activations and emotional states. In particular, a hyperplane classification was employed in the system for facial expressions representing major emotional states. The model’s primary advantage lies in its low computational complexity, which enables it to recognize changes in human emotional states through facial expressions without requiring specialized equipment, such as low-resolution or long-distance video cameras. The proposed approach is intended for use in control systems for various purposes, including security systems or monitoring drivers while operating vehicles. It was investigated that the proposed model could generate facial expressions similar to those produced by humans and that these expressions were recognized as conveying the intended emotional state by human observers. The authors also investigated the effect of different factors on the generation of facial expressions. Overall, the proposed model represents a promising approach for generating realistic facial expressions that mimic human emotional states and could have applications in improving security compliance in sensitive environments. However, carefully considering and managing potential ethical issues will be necessary to ensure the responsible use of this technology.

Facial expression generation based on variational AutoEncoder network and cloud computing

With the rapid development of portable terminal devices, such as smartphones, the development of expression generation technology makes the emotional communication between people in social networks more rich and diverse. However, there are still great challenges in deploying deep networks on mobile phones with limited resources. Moreover, the muscle movement of the face produces different facial expressions. To this end, this paper proposes an efficient emotion image generation framework based on Transformer and variational auto‐encoders (VAE) on the cloud platform. Specifically, we collect human face which is further transferred to the service. To exploit connections between muscles and expressions, we introduce the multi‐head attention mechanism to construct Transformer. To generate different emotional images, we design a novel emotion generation model named Transformer‐embedded VAE (TEVAE). All the experimental results show that the proposed TEVAE obtains higher performance in emotion image generation tasks compared to VAE.

Self-Supervised Spontaneous Latent-Based Facial Expression Sequence Generation

In this article, we investigate the spontaneity issue in facial expression sequence generation. Current leading methods in the field are commonly reliant on manually adjusted conditional variables to direct the model to generate a specific class of expression. We propose a neural network-based method which uses Gaussian noise to model spontaneity in the generation process, removing the need for manual control of conditional generation variables. Our model takes two sequential images as input, with additive noise, and produces the next image in the sequence. We trained two types of models: single-expression, and mixed-expression. With single-expression, unique facial movements of certain emotion class can be generated; with mixed expressions, fully spontaneous expression sequence generation can be achieved. We compared our method to current leading generation methods on a variety of publicly available datasets. Initial qualitative results show our method produces visually more realistic expressions and facial action unit (AU) trajectories; initial quantitative results using image quality metrics (SSIM and NIQE) show the quality of our generated images is higher. Our approach and results are novel in the field of facial expression generation, with potential wider applications to other sequence generation tasks.

Therapeutic strategy for facial paralysis based on the combined application of Si-based agent and methylcobalamin

Facial paralysis results in the decline in the generation of facial expressions and is attributed to several causes. Intractable facial paralysis has a poor prognosis, and new treatments are required. Facial paralysis results in the decline in the generation of facial expressions and is attributed to several causes. Reactive oxygen species can inhibit peripheral nerve regeneration after injury. Therefore, the administration of an appropriate antioxidant can promote nerve regeneration. Silicon (Si)-based agents can react with water to generate antioxidant hydrogen. Oral administration of Si-based agents can effectively alleviate symptoms of disease models associated with oxidative stress. Thus, we orally administered a Si-based agent to a facial paralysis model mice to investigate whether promotion of nerve regeneration occurred. The combined administration of methylcobalamin (MeCbl) with the Si-based agent was also investigated. The Si-based agent improved the clinical score evaluation of facial paralysis. Electroneuronography and immunostaining showed that the Si-based agent promoted myelination and recovery of facial nerve function. Furthermore, in the drug-administered group, oxidative stress associated with facial nerve injury was reduced more than that in the non-administered group. The clinical score evaluation, neuroregeneration effect, and reduction of oxidative stress were improved in the combination group compared to the single administration group. The Si-based agent could rapidly improve the disappearance of facial expressions by promoting myelin sheath formation and alleviating oxidative stress. Combination therapy with a Si-based agent and MeCbl should improve the prognosis and treatment of intractable facial paralysis.

Facial expression video generation based-on spatio-temporal convolutional GAN: FEV-GAN

Facial expression generation has always been an intriguing task for scientists and researchers all over the globe. In this context, we present our novel approach for generating videos of the six basic facial expressions. Starting from a single neutral facial image and a label indicating the desired facial expression, we aim to synthesize a video of the given identity performing the specified facial expression. Our approach, referred to as FEV-GAN (Facial Expression Video GAN), is based on Spatio-temporal Convolutional GANs, that are known to model both content and motion in the same network. Previous methods based on such a network have shown a good ability to generate coherent videos with smooth temporal evolution. However, they still suffer from low image quality and low identity preservation capability. In this work, we address this problem by using a generator composed of two image encoders. The first one is pre-trained for facial identity feature extraction and the second for spatial feature extraction. We have qualitatively and quantitatively evaluated our model on two international facial expression benchmark databases: MUG and Oulu-CASIA NIR&VIS. The experimental results analysis demonstrates the effectiveness of our approach in generating videos of the six basic facial expressions while preserving the input identity. The analysis also proves that the use of both identity and spatial features enhances the decoder ability to better preserve the identity and generate high-quality videos. The code and the pre-trained model will soon be made publicly available.

Facial Emotion Expressions in Human–Robot Interaction: A Survey

AbstractFacial expressions are an ideal means of communicating one’s emotions or intentions to others. This overview will focus on human facial expression recognition as well as robotic facial expression generation. In the case of human facial expression recognition, both facial expression recognition on predefined datasets as well as in real-time will be covered. For robotic facial expression generation, hand-coded and automated methods i.e., facial expressions of a robot are generated by moving the features (eyes, mouth) of the robot by hand-coding or automatically using machine learning techniques, will also be covered. There are already plenty of studies that achieve high accuracy for emotion expression recognition on predefined datasets, but the accuracy for facial expression recognition in real-time is comparatively lower. In the case of expression generation in robots, while most of the robots are capable of making basic facial expressions, there are not many studies that enable robots to do so automatically. In this overview, state-of-the-art research in facial emotion expressions during human–robot interaction has been discussed leading to several possible directions for future research.

Expression retargeting from images to three‐dimensional face models represented in texture space

AbstractFacial expressions play a very important role in creating a vivid and realistic virtual character. This paper proposes a convolutional neural network‐based facial expression retargeting framework named EXPUV‐Net, which can extract expression information from an image and transfer it to a specified 3D face. With the help of the nonlinear generator, the proposed framework can generate rich facial expressions and does not rely on the blendshape model. A data augmentation approach is adopted in this paper, which can improve the adaptability of the proposed framework. The framework can realize expression retargeting for face models with different topologies. This paper also demonstrates the effectiveness of the proposed framework through some experiments and compares it with some linear face model‐based methods for facial component extraction and face reconstruction. The results show that the proposed framework has better performance than the linear methods. The proposed framework is very helpful for facial expression generation for virtual characters.

Facial Expression Manipulation for Personalized Facial Action Estimation

Limited sizes of annotated video databases of spontaneous facial expression, imbalanced action unit labels, and domain shift are three main obstacles in training models to detect facial actions and estimate their intensity. To address these problems, we propose an approach that incorporates facial expression generation for facial action unit intensity estimation. Our approach reconstructs the 3D shape of the face from each video frame, aligns the 3D mesh to a canonical view, and trains a GAN-based network to synthesize novel images with facial action units of interest. We leverage the synthetic images to achieve two goals: 1) generating AU-balanced databases, and 2) tackling domain shift with personalized networks. To generate a balanced database, we synthesize expressions with varying AU intensities and perform semantic resampling. Our experimental results on FERA17 show that networks trained on synthesized facial expressions outperform those trained on actual facial expressions and surpass current state-of-the-art approaches. To tackle domain shift, we propose personalizing pretrained networks. We generate synthetic expressions of each target subject with varying AU intensity labels and use the person-specific synthetic images to fine-tune pretrained networks. To evaluate performance of the personalized networks, we use DISFA and PAIN databases. Personalized networks, which require only a single image from each target subject to generate synthetic images, achieved significant improvement in generalizing to unseen domains.

Acquiring identity and expression information from monocular face image

AbstractIn this paper, a flexible deep learning‐based framework is proposed that can extract expression and identity information from monocular images and can combine the extracted identity and expression from different images to generate new face models. In this framework, two encoders are used to extract expression and identity information, and three decoders are used to visualize the information by generating face models containing only expression, only identity, and fused expression and identity. By aligning the corresponding vertices of the parts with the same semantic on the face, an error evaluation method between face models with different topologies is proposed, which can more intuitively reflect the error distribution. The experimental results show that the proposed framework has higher accuracy than face component extraction by blendshape. The framework can be used for the facial expression generation of virtual humans, which is helpful for emotion transmission and language supplementation.

Data-driven Dimensional Expression Generation via Encapsulated Variational Auto-Encoders

Concerning facial expression generation, relying on the sheer volume of training data, recent advances on generative models allow high-quality generation of facial expressions free of the laborious facial expression annotating procedure. However, these generative processes have limited relevance to the psychological conceptualised dimensional plane, i.e., the Arousal-Valence two-dimensional plane, resulting in the generation of psychological uninterpretable facial expressions. For this, in this research, we seek to present a novel generative model, targeting learning the psychological compatible (low-dimensional) representations of facial expressions to permit the generation of facial expressions along the psychological conceptualised Arousal-Valence dimensions. To generate Arousal-Valence compatible facial expressions, we resort to a novel form of the data-driven generative model, i.e., the encapsulated variational auto-encoders (EVAE), which is consisted of two connected variational auto-encoders. Two harnessed variational auto-encoders in our EVAE model are concatenated with a tuneable continuous hyper-parameter, which bounds the learning of EVAE. Since this tuneable hyper-parameter, along with the linearly sampled inputs, largely determine the process of generating facial expressions, we hypothesise the correspondence between continuous scales on the hyper-parameter and sampled inputs, and the psychological conceptualised Arousal-Valence dimensions. For empirical validations, two public released facial expression datasets, e.g., the Frey faces and FERG-DB datasets, were employed here to evaluate the dimensional generative performance of our proposed EVAE. Across two datasets, the generated facial expressions along our two hypothesised continuous scales were observed in consistent with the psychological conceptualised Arousal-Valence dimensions. Applied our proposed EVAE model to the Frey faces and FERG-DB facial expression datasets, we demonstrate the feasibility of generating facial expressions along with the conceptualised Arousal-Valence dimensions. In conclusion, to generate facial expressions along the psychological conceptualised Arousal-Valance dimensions, we propose a novel type of generative model, i.e., encapsulated variational auto-encoders (EVAE), allowing the generation process to be disentangled into two tuneable continuous factors. Validated in two publicly available facial expression datasets, we demonstrate the association between these factors and Arousal-Valence dimensions in facial expression generation, deriving the data-driven Arousal-Valence plane in affective computing. Despite its embryonic stage, our research may shed light on the prospect of continuous, dimensional affective computing.

ExGenNet: Learning to Generate Robotic Facial Expression Using Facial Expression Recognition.

The ability of a robot to generate appropriate facial expressions is a key aspect of perceived sociability in human-robot interaction. Yet many existing approaches rely on the use of a set of fixed, preprogrammed joint configurations for expression generation. Automating this process provides potential advantages to scale better to different robot types and various expressions. To this end, we introduce ExGenNet, a novel deep generative approach for facial expressions on humanoid robots. ExGenNets connect a generator network to reconstruct simplified facial images from robot joint configurations with a classifier network for state-of-the-art facial expression recognition. The robots’ joint configurations are optimized for various expressions by backpropagating the loss between the predicted expression and intended expression through the classification network and the generator network. To improve the transfer between human training images and images of different robots, we propose to use extracted features in the classifier as well as in the generator network. Unlike most studies on facial expression generation, ExGenNets can produce multiple configurations for each facial expression and be transferred between robots. Experimental evaluations on two robots with highly human-like faces, Alfie (Furhat Robot) and the android robot Elenoide, show that ExGenNet can successfully generate sets of joint configurations for predefined facial expressions on both robots. This ability of ExGenNet to generate realistic facial expressions was further validated in a pilot study where the majority of human subjects could accurately recognize most of the generated facial expressions on both the robots.

Total Generate: Cycle in Cycle Generative Adversarial Networks for Generating Human Faces, Hands, Bodies, and Natural Scenes

We propose a novel and unified Cycle In Cycle Generative Adversarial Network (C2GAN) for generating human faces, hands, bodies, and natural scenes. Our proposed C2GAN is a cross-modal model exploring a joint exploitation of the input image data and guidance data in an interactive manner. C2GAN contains two different generators, i.e., an image-generation generator and a guidance-generation generator. Both generators are mutually connected and trained in an end-to-end fashion and explicitly form three cycled subnets, i.e., one image generation cycle and two guidance generation cycles. Each cycle aims at reconstructing the input domain and simultaneously produces a useful output involved in the generation of another cycle. In this way, the cycles constrain each other implicitly providing complementary information from both image and guidance modalities and bringing an extra supervision gradient across the cycles, facilitating a more robust optimization of the whole model. Extensive experimental results on four guided image-to-image translation subtasks, i.e., person image generation, facial expression generation, hand gesture-to-gesture translation, and cross-view image translation, demonstrate that the proposed C2GAN is effective in generating more realistic images compared with state-of-the-art models.

Laughter and smiling facial expression modelling for the generation of virtual affective behavior.

Laughter and smiling are significant facial expressions used in human to human communication. We present a computational model for the generation of facial expressions associated with laughter and smiling in order to facilitate the synthesis of such facial expressions in virtual characters. In addition, a new method to reproduce these types of laughter is proposed and validated using databases of generic and specific facial smile expressions. In particular, a proprietary database of laugh and smile expressions is also presented. This database lists the different types of classified and generated laughs presented in this work. The generated expressions are validated through a user study with 71 subjects, which concluded that the virtual character expressions built using the presented model are perceptually acceptable in quality and facial expression fidelity. Finally, for generalization purposes, an additional analysis shows that the results are independent of the type of virtual character’s appearance.