3D Facial Expression Research Articles

Barrier Function to Skin Elasticity in Talking Head

In this paper, we target the problem of generating facial expressions from a piece of audio. This is challenging since both audio and video have inherent characteristics that are distinct from the other. Some words may have identical lip movements, and speech impediments may prevent lip-reading in some individuals. Previous approaches to generating such a talking head suffered from stiff expressions. This is because they focused only on lip movements and the facial landmarks did not contain the information flow from the audio. Hence, in this work, we employ spatio-temporal independent component analysis to accurately sync the audio with the corresponding face video. Proper word formation also requires control over the face muscles that can be captured using a barrier function. We first validated the approach on the diffusion of salt water in coastal areas using a synthetic finite element simulation. Next, we applied it to 3D facial expressions in toddlers for which training data is difficult to capture. Prior knowledge in the form of rules is specified using Fuzzy logic, and multi-objective optimization is used to collectively learn a set of rules. We observed significantly higher F-measure on three real-world problems.

An Empirical Study of Portrait Rights on Emotional Evolution in Virtual Social Scenarios by Transformer and Cloud Computing

Facial expression recognition technology can improve the possibility of fugitives and persons subject to execution being discovered by public security and judicial personnel in the virtual world, thereby improving the actual effectiveness and credibility of the law. There may also be a risk of portrait rights infringement with this technology, and the user of the technology needs to inform the person whose facial expression is being extracted in advance and obtain permission. With the rapid progression of deep learning and artificial intelligence, 3D facial expression modeling has garnered increased significance in computer vision and graphics. We propose an innovative approach combining Transformer models with cloud computing to simulate facial expression evolution in virtual social environments. Leveraging Transformer-based feature extraction, our model integrates emotional cues from various modalities to accurately capture subtle changes over time.

Non-corresponding and topology-free 3D face expression transfer

Non-corresponding and topology-free 3D face expression transfer

3D Facial Expression Recognition Based on Geometric Feature Fusion

In order to overcome the problems of missing important information and di-mensionality disaster of three-dimensional (3D) facial expression, a novel method based on geometric feature fusion is proposed. This approach initially extracts distance feature vectors and angle feature vectors from key facial regions such as the eyes, mouth, and eyebrows using direct geometric feature. Subsequently, the K-nearest neighbor (K-NN) algorithm is used to obtain the distance feature vector candidate sets and angle feature vector candidate sets separately. Finally, the maximum-minimum rule is utilized to fuse these candidate sets into a single feature vector, completing the recognition of 3D facial expression. Experimental results on the BU-3DFE database demonstrate that this method achieves on overall recognition rate of 97.6%, exhibiting excellent robustness to varia-tions in facial expressions. Furthermore, this approach can serve as a valuable reference for future research in the field of 3D face recognition.

Depression Level Assessment based on 3D CNN and Facial Expression Videos

Depression Level Assessment based on 3D CNN and Facial Expression Videos

4D Facial Expression Diffusion Model

Facial expression generation is one of the most challenging and long-sought aspects of character animation, with many interesting applications. The challenging task, traditionally having relied heavily on digital craftspersons, remains yet to be explored. In this paper, we introduce a generative framework for generating 3D facial expression sequences (i.e. 4D faces) that can be conditioned on different inputs to animate an arbitrary 3D face mesh. It is composed of two tasks: (1) Learning the generative model that is trained over a set of 3D landmark sequences, and (2) Generating 3D mesh sequences of an input facial mesh driven by the generated landmark sequences. The generative model is based on a Denoising Diffusion Probabilistic Model (DDPM), which has achieved remarkable success in generative tasks of other domains. While it can be trained unconditionally, its reverse process can still be conditioned by various condition signals. This allows us to efficiently develop several downstream tasks involving various conditional generation, by using expression labels, text, partial sequences, or simply a facial geometry. To obtain the full mesh deformation, we then develop a landmark-guided encoder-decoder to apply the geometrical deformation embedded in landmarks on a given facial mesh. Experiments show that our model has learned to generate realistic, quality expressions solely from the dataset of relatively small size, improving over the state-of-the-art methods. Videos and qualitative comparisons with other methods can be found at https://github.com/ZOUKaifeng/4DFM. Code and models will be made available upon acceptance.

CalD3r and MenD3s: Spontaneous 3D facial expression databases

In the last couple of decades, the research on 3D facial expression recognition has been fostered by the creation of tailored databases containing prototypical expressions of different individuals and by the advances in cost effective acquisition technologies. Though, most of the currently available databases consist of exaggerated facial expressions, due to the imitation principle which they rely on. This makes these databases only partially employable for real world applications such as human-computer interaction for smart products and environments, health, and industry 4.0, as algorithms learn on these ‘inflated’ data which do not respond to ecological validity requirements. In this work, we present two novel 2D + 3D spontaneous facial expression databases of young adults with different geographical origin, in which emotions have been evoked thanks to affective images of the acknowledged IAPS and GAPED databases, and verified with participants’ self-reports. To the best of our knowledge, these are the first three-dimensional facial databases with emotions elicited by validated affective stimuli.

Free-HeadGAN: Neural Talking Head Synthesis With Explicit Gaze Control

We present Free-HeadGAN, a person-generic neural talking head synthesis system. We show that modeling faces with sparse 3D facial landmarks is sufficient for achieving state-of-the-art generative performance, without relying on strong statistical priors of the face, such as 3D Morphable Models. Apart from 3D pose and facial expressions, our method is capable of fully transferring the eye gaze, from a driving actor to a source identity. Our complete pipeline consists of three components: a canonical 3D key-point estimator that regresses 3D pose and expression-related deformations, a gaze estimation network and a generator that is built upon the architecture of HeadGAN. We further experiment with an extension of our generator to accommodate few-shot learning using an attention mechanism, in case multiple source images are available. Compared to recent methods for reenactment and motion transfer, our system achieves higher photo-realism combined with superior identity preservation, while offering explicit gaze control.

4D facial analysis: A survey of datasets, algorithms and applications

Facial information plays an important role in human communication, e.g., rich and nuanced facial expressions effectively convey emotions. Traditional data such as 2D facial images and videos are susceptible to perturbation from lighting and occlusion, while 3D static data such as mesh models lack the temporal information which is necessary to describe facial dynamics. Therefore, researches on 4D facial data (3D facial models together with time as the fourth dimension) have received considerable attention in recent years. 4D data can simultaneously reflect the complex facial temporal and space information. To fully explore these characteristics, we present a systematic overview of the 4D facial research in this paper. We give a review in terms of historical development of 4D facial datasets, acquisition process of these datasets, related algorithms and applications, and discussion of outstanding issues. We also analyze the 4D facial research works by summarizing and comparing them; in particular, we present the results on three kinds of tasks conducted on the 4D facial datasets, i.e., expression-related tasks (AU detection, expression recognition and retrieval), generation tasks (3D facial reconstruction, facial expression and facial animation generation), and other tasks (facial registration, facial recognition and facial disease diagnosis). Finally, we summarize the future research directions.

Influence of 3D characters image transfer on animation drawing based on detail feature preservation

Abstract Under the multiple driving of capital, new media and consumer group, the output value of Chinese animation industry keeps growing rapidly. Meanwhile, the quality of animation characters image determines the overall development level of the animation industry. This paper proposed a detail feature preservation-based 3D facial expression transfer method aiming at the design of 3D characters. By extracting the detail features of 3D facial model, the basic expression after removing the details was obtained. By using the improved joint learning method, the basic expression of the source model was transferred to the target model, and the feature modulus based on the Laplace factor was constructed. Finally, through the detail feature vector adjustment strategy, the target model with source basic expression was performed detail restoration. The multiple experiments show that the method proposed in this paper can transfer the expression of the source model to the target model without damage, and at the same time, preserve the personality detail features of the target model. Moreover, the animation expressions generated are real and natural, and the facial micro-movements of animation characters are rich, which positively affects the development of Chinese animation industry.

Correction to: 3D facial expression retargeting framework based on an identity-independent expression feature vector

Correction to: 3D facial expression retargeting framework based on an identity-independent expression feature vector

3D facial expression retargeting framework based on an identity-independent expression feature vector

3D facial expression retargeting framework based on an identity-independent expression feature vector

Where Are the Dots: Hardening Face Authentication on Smartphones With Unforgeable Eye Movement Patterns

With the ubiquitous adoption, mobile face authentication systems have been facing constant security challenges, particularly the spoofing risks. Except for those using specialized hardware, existing proposals for face anti-spoofing on mainstream smartphones either leverage people’s 3D face characteristics or various facial expressions. While showing progress towards more resilient face authentication, they are still vulnerable to recent advanced attacks (e.g., 3D mask attacks, video attacks, etc.). This paper presents GazeGuard, an on-device face anti-spoofing system that leverages unpredictable and unforgeable eye movement patterns to provide strong security guarantees against all known attacks. Targeting mainstream smartphones, GazeGuard is designed to conduct eye movement-based authentication using only 2D front cameras. Specifically, by presenting a series of short-lasting random dots on the screen (named gazecode), GazeGuard simultaneously captures a user’s gaze responses and the corresponding deformed periocular features to ensure both the freshness and correctness for the anti-spoofing face authentication. We have extensively tested GazeGuard’s performance over 50 volunteers. Using a 4-digit gazecode (just four random dots), GazeGuard achieves an average 90.39% authentication accuracy and 81.57 out of 100 System Usability Scale (SUS) scores. Under the same settings, GazeGuard achieves detection accuracy of 95.72% for image attack, 95.59% for video attack, 99.73% for 3D mask attack, and 100% for physical adversarial attack.

3D Virtual Garment Simulation and Facial Expression Recognition Technologies, Beauty Filter-based Appealing Self-presentation and Generative Artificial Intelligence Virtual Try-on Tools, and Deep Learning-based Gesture Recognition and Augmented Reality Face Makeup Systems for Body-Image Affective Judgments and Dysfunctional Appearance Beliefs

3D Virtual Garment Simulation and Facial Expression Recognition Technologies, Beauty Filter-based Appealing Self-presentation and Generative Artificial Intelligence Virtual Try-on Tools, and Deep Learning-based Gesture Recognition and Augmented Reality Face Makeup Systems for Body-Image Affective Judgments and Dysfunctional Appearance Beliefs

3D Face Reconstruction and Gaze Tracking in the HMD for Virtual Interaction

With the rapid development of virtual reality (VR) technology, VR headsets, a.k.a. Head-Mounted Displays (HMDs), are widely available, allowing immersive 3D content to be viewed. A natural need for truly immersive VR is to allow bidirectional communication: the user should be able to interact with the virtual world using facial expressions and eye gaze, in addition to traditional means of interaction. The typical application scenario includes VR virtual conferencing and virtual roaming, where ideally users are able to see other users expressions and have eye contact with them in the virtual world. In addition, eye gaze also provides a natural means of interaction with virtual objects. Despite significant achievements in recent years for reconstruction of 3D faces from RGB or RGB-D images, it remains a challenge to reliably capture and reconstruct 3D facial expressions including eye gaze when the user is wearing an HMD, because the majority of the face is occluded, especially those areas around the eyes which are essential for recognizing facial expressions and eye gaze. In this paper, we introduce a novel real-time system that is able to capture and reconstruct 3D faces wearing HMDs, and robustly recover eye gaze. We further propose a novel method to map eye gaze directions to the 3D virtual world, which provides a novel and useful interactive mode in VR. We compare our method with state of-the-art techniques both qualitatively and quantitatively, and demonstrate the effectiveness of our system using live capture.

The effect of pedestrian lighting on facial expression recognition with 3D models: A lab experiment

One function of road lighting is to enable pedestrians to evaluate the intentions of other people or to make interpersonal judgement. This paper aims to investigate the effect of lighting conditions on the ability of pedestrian to recognize facial expressions of 3D models. Fifteen lighting conditions were examined: five vertical illuminances (0.33, 1.00, 3.33, 10.0, and 30.0 lx) crossing spectral power distribution with three different scotopic/photopic ratios (0.57, 1.77 and 25.13). The results show that recognition of 3D facial expressions tended to be improved as illuminance increased. However, manipulation on spectral power distribution would not lead to better performance on 3D facial expression recognition. Compared to previous studies that used 2D images as visual stimuli, the current results from the 3D model present a slightly better performance on facial expression recognition. These findings are expected to contribute to the recommendations of optimal lighting levels based on facial expression recognition, and provide further evidence for the development of lighting standards.

Facial Expression Recognition with Geometric Scattering on 3D Point Clouds.

As one of the pioneering data representations, the point cloud has shown its straightforward capacity to depict fine geometry in many applications, including computer graphics, molecular structurology, modern sensing signal processing, and more. However, unlike computer graphs obtained with auxiliary regularization techniques or from syntheses, raw sensor/scanner (metric) data often contain natural random noise caused by multiple extrinsic factors, especially in the case of high-speed imaging scenarios. On the other hand, grid-like imaging techniques (e.g., RGB images or video frames) tend to entangle interesting aspects with environmental variations such as pose/illuminations with Euclidean sampling/processing pipelines. As one such typical problem, 3D Facial Expression Recognition (3D FER) has been developed into a new stage, with remaining difficulties involving the implementation of efficient feature abstraction methods for high dimensional observations and of stabilizing methods to obtain adequate robustness in cases of random exterior variations. In this paper, a localized and smoothed overlapping kernel is proposed to extract discriminative inherent geometric features. By association between the induced deformation stability and certain types of exterior perturbations through manifold scattering transform, we provide a novel framework that directly consumes point cloud coordinates for FER while requiring no predefined meshes or other features/signals. As a result, our compact framework achieves 78.33% accuracy on the Bosphorus dataset for expression recognition challenge and 77.55% on 3D-BUFE.

Jointly Optimizing Expressional and Residual Models for 3D Facial Expression Removal

This article proposes a facial expression removal method to recover a 3D neutral face from a single 3D expressional or non-neutral face. We treat a 3D non-neutral face as the sum of its neutral one and the residual. This can be satisfied if the correspondence between 3D vertices of expressional faces and those of neutral faces is established. We propose a non-rigid deformation method to establish the correspondence between 3D faces. Then, according to algebra inequality, the minimization of a neutral face model can be replaced by the minimization of its upper bound, i.e., the errors of an expressional face model and a residual model. Thus, we co-optimize the representation errors of the latter two models and build the relationship between the representation coefficients of the two models. Given an expressional face as the input, its corresponding neutral face can be inferred by the associative representation parameters in these two models. In the testing stage, we use an iterative joint fitting scheme to obtain a more accurate recovery. Extensive experiments are conducted to evaluate our method. The results show that our method obtains considerably better performance than existing methods in terms of average root mean square errors and recognition rates, and also better visual effects.

Low Rank Graph Regularization Embedding for 2D+3D Facial Expression Recognition

In this paper, a novel low rank graph regularization embedding for 3D facial expression recognition (LRGREFER) approach is proposed, in which the core tensor is utilized to characterize the low-rank attribute among the samples combined with the factor matrices with the graph regularization embedding. At first, a model based on a 4D tensor is constructed from the facial expression data. By Tucker decomposing the constructed 4D tensor, a resulting core tensor and factor matrices in different tensor modes are utilized to characterize the low-rankness among samples. Because of the loss of information during modelling the 4D tensor, the missing data from partly observed facial expression data are recovered by embedding the tensor completion. Finally, the proposed model is handled and solved by adopting the alternating direction method of multipliers (ADMM). Meanwhile, the classification prediction of facial expressions are implemented by Multi-class-SVM. Numerical experiments are conducted on BU-3DFE database. The experiment results have been verified that our proposed approach is more competitive.

Low Rank Tucker Decomposition for 2D+3D Facial Expression Recognition

In this paper, a novel low rank Tucker decomposition for 3D facial expression recognition (LRTDFER) approach is proposed, in which the core tensor is utilized to characterize the low-rank attribute among the samples combined with the factor matrices with the graph regularization embedding. At first, a model based on a 4D tensor is constructed from the facial expression data. By Tucker decomposing the constructed 4D tensor, a resulting core tensor and factor matrices in different tensor modes are utilized to characterize the low-rankness among samples. Because of the loss of information during modelling the 4D tensor, the missing data from partly observed facial expression data are recovered by embedding the tensor completion. Finally, the proposed model is handled and solved by adopting the alternating direction method of multipliers (ADMM). Meanwhile, the classification prediction of facial expressions are implemented by Multi-class-SVM. Numerical experiments are conducted on BU-3DFE database. The experiment results have been verified that our proposed approach is more competitive.