Dynamic Facial Expression Recognition Research Articles

Observe finer to select better: Learning key frame extraction via semantic coherence for dynamic facial expression recognition in the wild

In dynamic facial expression sequences captured in natural environments, the number and distribution of expression peak frames (key frames) show significant variation within video sequences. Efficient extraction and utilization of these key expression frames are crucial for improving the accuracy of expression recognition. To address the above challenges, this paper proposed the Key Frame Extraction via Semantic Coherence (KFE-SC) method. The method explores the spatial-temporal emotional semantic information within sequences, employing a coarse-to-fine approach to extract key expression frames, thus facilitating classification tasks. By introduce a Spatial Semantic Coherence (SSC) module and designing a Global-Local Feature Aggregation (GLFA) unit, the method explores the spatial semantic correlations at global-local levels in images. Through calculating weight proportions, the method achieves a coarse-grained extraction of key frames in the spatial dimension. Considering that video sequences contain both spatial and temporal semantic relevance, KFE-SC further introduces a Temporal Semantic Coherence (TSC) module, which include the Temporal Alignment Aggregation (TA2) unit to align short and long-term temporal information and explore temporal semantic relevance within the sequences, enabling fine-grained extraction of key frames in the temporal dimension through weight calculations.

The investigation of traditional models and machine learning models in dynamic facial expression recognition

In everyday life, dynamic facial expressions are merely continuous human responses to external events. However, in human-computer interaction, rapidly recognizing changes in facial expressions from video streams is a relatively complex process. This complexity renders Dynamic Facial Expression Recognition (DFER) a critical research task in the domains of computer vision and image processing. This paper analyses the correlations and contrasts between static and dynamic facial expression research, highlighting key issues in the study of dynamic facial expressions, such as dynamic feature extraction and frame extraction. After that, it enumerates significant algorithms in both traditional models and deep learning models, providing an analysis of the advantages and disadvantages of these two major approaches. At the same time, it investigates the reasons behind the transition of research models for DFER from traditional methods to deep learning approaches. The paper focuses on two notable models from each approach: Histogram of Oriented Gradient (HOG) for processing raw images, Support Vector Machine (SVM) for data classification in traditional models. Convolutional Neural Network (CNN) for spatial feature extraction and Long Short-Term Memory (LSTM) for temporal feature extraction in deep learning models. These models are discussed in detail concerning their strengths and weaknesses, operational processes, and performance outcomes. In the concluding section, the author summarizes the main factors influencing research in this field and the current challenges encountered. By focusing on future research directions, the paper also presents a review of recent methodologies and offers insightful research directions for further investigation.

A joint local spatial and global temporal CNN-Transformer for dynamic facial expression recognition

Unlike conventional video action recognition, Dynamic Facial Expression Recognition (DFER) tasks exhibit minimal spatial movement of objects. Addressing this distinctive attribute, we propose an innovative CNN-Transformer model, named LSGTNet, specifically tailored for DFER tasks. Our LSGTNet comprises three stages, each composed of a spatial CNN (Spa-CNN) and a temporal transformer (T-Former) in sequential order. The Spa-CNN extracts spatial features from images, yielding smaller-sized feature maps to alleviate the computational complexity for subsequent T-Former. The T-Former integrates global temporal information from the same spatial positions across different time frames while retaining the feature map dimensions. The alternating interplay between Spa-CNN and T-Former ensures a continuous fusion of spatial and temporal information, leading our model to excel across various real-world datasets. To the best of our knowledge, this is the first method to address the DFER challenge by focusing on capturing the temporal changes in muscles within local spatial regions. Our method has achieved state-of-the-art results on multiple in-the-wild datasets and datasets under laboratory conditions.

Transformer-Based Multimodal Emotional Perception for Dynamic Facial Expression Recognition in the Wild

Transformer-Based Multimodal Emotional Perception for Dynamic Facial Expression Recognition in the Wild

LSTPNet: Long short-term perception network for dynamic facial expression recognition in the wild

In-the-wild dynamic facial expression recognition (DFER) is a very challenging task, and previous methods based on convolutional neural networks (CNNs), recurrent neural networks (RNNs), or Transformers emphasize the extraction of either short-term temporal information or long-term temporal information from facial video sequences. Different from existing methods, this paper proposes a long short-term perceptimon network (LSTPNet) for dynamic facial expression recognition, taking into account the joint perception of the above two temporal cues to benefit the DFER task. Specifically, we propose a long short-term temporal Transformer (LSTformer) which can perceive both long-term and short-term temporal information effectively. In addition, we introduce a temporal channel excitation (TCE) module extended from the previous notable efficient channel attention (ECA) module, in order to establish temporal attention for intermediate features within the backbone network, and obtain more temporally representative features. Experimental results on three benchmark datasets demonstrate the state-of-the-art performance of the proposed LSTPNet. The code will be available at https://github.com/LLFabiann/LSTPNet/.

Recognizing facial expressions of emotion amid noise: A dynamic advantage.

Humans communicate internal states through complex facial movements shaped by biological and evolutionary constraints. Although real-life social interactions are flooded with dynamic signals, current knowledge on facial expression recognition mainly arises from studies using static face images. This experimental bias might stem from previous studies consistently reporting that young adults minimally benefit from the richer dynamic over static information, whereas children, the elderly, and clinical populations very strongly do (Richoz, Jack, Garrod, Schyns, & Caldara, 2015, Richoz, Jack, Garrod, Schyns, & Caldara, 2018b). These observations point to a near-optimal facial expression decoding system in young adults, almost insensitive to the advantage of dynamic over static cues. Surprisingly, no study has yet tested the idea that such evidence might be rooted in a ceiling effect. To this aim, we asked 70 healthy young adults to perform static and dynamic facial expression recognition of the six basic expressions while parametrically and randomly varying the low-level normalized phase and contrast signal (0%-100%) of the faces. As predicted, when 100% face signals were presented, static and dynamic expressions were recognized with equal efficiency with the exception of those with the most informative dynamics (i.e., happiness and surprise). However, when less signal was available, dynamic expressions were all better recognized than their static counterpart (peaking at ∼20%). Our data show that facial movements increase our ability to efficiently identify emotional states of others under the suboptimal visual conditions that can occur in everyday life. Dynamic signals are more effective and sensitive than static ones for decoding all facial expressions of emotion for all human observers.

Empower smart cities with sampling-wise dynamic facial expression recognition via frame-sequence contrastive learning

In the construction of smart cities, facial expression analysis plays a crucial role. It can be used in traffic monitoring systems to alleviate traffic pressure by analyzing the emotional states of drivers and passengers. In the field of smart healthcare, it can provide more precise treatment and services to patients. In the realm of social entertainment, it can offer more intelligent and personalized interactions. In summary, the application of emotion computing technology will play an increasingly significant role in the development of smart cities in the future. In the task of dynamic facial expression recognition (DFER), analyzing the spatial–temporal features of video sequences has become a common research approach. However, facial expression sequences often contain a significant number of neutral frames and noisy frames, potentially increasing computational costs and reducing performance. Effectively extracting key frames for spatial–temporal feature analysis is a critical aspect of dynamic facial expression recognition. To address this issue, we proposed a sampling-wise dynamic facial expression recognition via frame-Sequence contrastive learning method, called SW-FSCL. The SW-FSCL method aims to improve the performance of DFER by using intelligent dual-stream sampling strategies and frame-sequence contrastive learning, extract key frame and reduce the impact of neutral frames and noisy frames. We proposed a key frame proposal (KFP) block to analyze the spatial–temporal features of sequences, calculating weight ratios for key frame extraction. Due to potential information loss in long sequences, we introduce a temporal aggregation (TA) block to prevent data loss and ensure the integrity of temporal information. The experimental results provide compelling evidence that the proposed approach not only outperforms all current state-of-the-art algorithms on two widely-used benchmark datasets (DFEW, FERV39k), but also visualization results produces insights into the interpretability of the SW-FSCL method.

A Survey of Deep Learning-based Facial Expression Recognition Research

Facial expression is one of the ways to convey emotional expression. Deep learning is used to analyze facial expression to understand people's true feelings, and human-computer interaction is integrated into it. However, in the natural real environment and various interference (such as lighting, age and ethnicity), facial expression recognition will face many challenges. In recent years, with the development of artificial intelligence, scholars have studied more and more facial expression recognition in the case of interference, which not only promotes the theoretical research, but also makes it popularized in the application. Facial expression recognition is to identify facial expressions to carry out emotion analysis, and emotion analysis can be analyzed with the help of facial expressions, speech, text, video and other signals. Therefore, facial expression recognition can be regarded as a research direction of emotion analysis. This paper focuses on the perspective of facial expression recognition to summarize. In the process of facial expression recognition, researchers usually try to combine multiple modal information such as voice, text, picture and video for analysis. Due to the differences between single-modal data set and multi-modal data set, this paper will analyze static facial expression recognition, dynamic facial expression recognition and multi-modal fusion. This research has a wide range of applications, such as: smart elderly care, medical research, detection of fatigue driving and other fields.

Enhanced spatial-temporal learning network for dynamic facial expression recognition

The recognition of dynamic facial expressions has received increasing attention since they can better reflect the real expression process of emotion than a static image. However, due to various factors such as subtle variation differences, pose, occlusion, and illumination, it has been a challenging vision task to obtain discriminative expression features in dynamic facial expression recognition. Traditional CNN-based deep learning networks lack global and temporal contextual expression understanding, which tends to affect the final recognition of dynamic expressions. Therefore, we propose an enhanced spatial–temporal learning network (ESTLNet) for more robust dynamic facial expression recognition, which consists of a spatial fusion learning module (SFLM) and a temporal transformer enhancement module (TTEM). First, the SFLM obtains a more expressive spatial feature representation through a dual-channel feature fusion learning module. Then, the TTEM extracts more valid temporal contextual expression features based on the above spatial features through an encoder constructed by cascading a self-attention learning network and an effective gated feed-forward network. Finally, the co-enhanced spatial–temporal model approach is assessed on the four broadly used dynamic expression datasets (DFEW, AFEW, CK+, and Oulu-CASIA). Extensive experimental outcomes demonstrate that our approach surpasses several existing state-of-the-art methods, leading to notable enhancements in performance.

Prosopagnosia elicits atypical fixation patterns during dynamic facial expression recognition

Prosopagnosia elicits atypical fixation patterns during dynamic facial expression recognition

Quantifying dynamic facial expression recognition thresholds in prosopagnosia

Quantifying dynamic facial expression recognition thresholds in prosopagnosia

Dynamic facial expression recognition with pseudo‐label guided multi‐modal pre‐training

AbstractDue to the huge cost of manual annotations, the labelled data may not be sufficient to train a dynamic facial expression (DFR) recogniser with good performance. To address this, the authors propose a multi‐modal pre‐training method with a pseudo‐label guidance mechanism to make full use of unlabelled video data for learning informative representations of facial expressions. First, the authors build a pre‐training dataset of videos with aligned vision and audio modals. Second, the vision and audio feature encoders are trained through an instance discrimination strategy and a cross‐modal alignment strategy on the pre‐training data. Third, the vision feature encoder is extended as a dynamic expression recogniser and is fine‐tuned on the labelled training data. Fourth, the fine‐tuned expression recogniser is adopted to predict pseudo‐labels for the pre‐training data, and then start a new pre‐training phase with the guidance of pseudo‐labels to alleviate the long‐tail distribution problem and the instance‐class confliction. Fifth, since the representations learnt with the guidance of pseudo‐labels are more informative, a new fine‐tuning phase is added to further boost the generalisation performance on the DFR recognition task. Experimental results on the Dynamic Facial Expression in the Wild dataset demonstrate the superiority of the proposed method.

Intensity-Aware Loss for Dynamic Facial Expression Recognition in the Wild

Compared with the image-based static facial expression recognition (SFER) task, the dynamic facial expression recognition (DFER) task based on video sequences is closer to the natural expression recognition scene. However, DFER is often more challenging. One of the main reasons is that video sequences often contain frames with different expression intensities, especially for the facial expressions in the real-world scenarios, while the images in SFER frequently present uniform and high expression intensities. Nevertheless, if the expressions with different intensities are treated equally, the features learned by the networks will have large intra-class and small inter-class differences, which are harmful to DFER. To tackle this problem, we propose the global convolution-attention block (GCA) to rescale the channels of the feature maps. In addition, we introduce the intensity-aware loss (IAL) in the training process to help the network distinguish the samples with relatively low expression intensities. Experiments on two in-the-wild dynamic facial expression datasets (i.e., DFEW and FERV39k) indicate that our method outperforms the state-of-the-art DFER approaches. The source code will be available at https://github.com/muse1998/IAL-for-Facial-Expression-Recognition.

Cross-view adaptive graph attention network for dynamic facial expression recognition

Cross-view adaptive graph attention network for dynamic facial expression recognition

HiT-MST: Dynamic facial expression recognition with hierarchical transformers and multi-scale spatiotemporal aggregation

Facial expression recognition rarely explores complex spatiotemporal dependencies among facial regions at different scales. This paper proposes a transformer-based three-layer hierarchical architecture that incorporates multi-scale spatiotemporal aggregation for dynamic facial expression recognition. The hierarchical structure consists of bottom-to-top layers, each comprising transformer encoders with local self-attention mechanisms. These encoders gradually expand their receptive fields through hierarchical spatiotemporal aggregation, enabling the modeling of spatiotemporal context dependencies among facial regions at different scales and across consecutive frames. Consequently, the bottom-to-top layers correspond to learning the fine-grained, coarse-grained, and global facial representations. To evaluate the performance of our proposed framework, we conducted extensive experiments on four public datasets. The comparison results demonstrate that our proposed framework outperforms the state-of-the-art, with accuracies of 79.09%, 62.19%, 64.85%, and 59.79% on the RML, eNTERFACE'05, RAVDESS, and AFEW datasets, respectively. Ablation experiments, statistical significance tests, and visualization analyses indicate that the proposed framework successfully learns emotional-salient facial representations.

Spatio-temporal modelling with multi-gradient features and elongated quinary pattern descriptor for dynamic facial expression recognition

We propose a new spatio-temporal modelling approach for Dynamic Facial Expression Recognition (DFER). We first convert the domain of the spatial images in the sequence to the gradient of magnitude and angle images at different orientations. Robust gradient components are developed to deal with difficult types of illuminations, such as darkness, by forming the eight edge responses of the Gaussian mask. To describe the dynamic Facial Expression (FE) changes we extend the Elongated Quinary Pattern (EQP) descriptor to encode separately the anisotropic structure of the uniform patterns from Three Orthogonal Planes (TOP) of each gradient sequence. Then each encoded sequence is divided into a stack of block volumes in the XY, XT and YT planes. For each plane, the co-occurrence of histogram features are calculated from each block volume and concatenated together. Simple three-dimensional histogram features are generated by concatenating the histogram features of all planes. A Multi Classifier System (MCS) based on a multi-class Support Vector Machine (SVM) is adopted to combine all scores for the encoded sequences. The proposed approach is evaluated with the challenging MMI and Oulu-CASIA databases with different set-ups and advantage has been shown in terms of generalisation to different databases, together with robustness against difficult pose variations and illumination changes. In terms of Recognition Accuracy (RA), a comparison is established with DFER methods in the literature. A high recognition rate of 79.23% is attained in the case of six classes when applied to the MMI database which surpasses all the state-of-the-art results.

Phase Space Reconstruction Driven Spatio-Temporal Feature Learning for Dynamic Facial Expression Recognition

Automatic Dynamic Facial Expression Recognition (DFER) is a challenging task, since how to effectively capture facial temporal dynamics is still an open problem. In this article, we regard variations of facial expressions as a dynamic system in accord with certain rules, and try to explore the fundamental temporal properties for recognizing dynamic expressions. Inspired by the phase space reconstruction method for time series analysis, we propose a novel network named Phase Space Reconstruction Network (PSRNet) for learning spatio-temporal features of facial expressions. First, 3D convolutional neural networks are used to extract spatial and short-term temporal features, which indicate the state of each frame and are termed as observations in the phase space. All the observations compose the trajectory of the dynamical system. Then, a data-driven across-correlation matrix is inferred to reveal the relationship of the observations. With this matrix, the phase space reconstruction module reconstructs the trajectory by aggregating the observations adaptively in the phase space. Reconstructed observations represent the gradual process of dynamic facial expressions, which is beneficial to recognize these expressions. The experiment results on three databases (Oulu, MMI, and CK+) demonstrate that the proposed PSRNet can extract more informative and representative spatio-temporal features for DFER. Moreover, the visualization of intermediate features reveals that the reconstructed features have global consistency in facial regions and the underlying evolutionary pattern of dynamic facial expression.

Facial expression recognition based on deep learning

Background and objectiveFacial expression recognition technology will play an increasingly important role in our daily life. Autonomous driving, virtual reality and all kinds of robots integrated into our life depend on the development of facial expression recognition technology. Many tasks in the field of computer vision are based on deep learning technology and convolutional neural network. The paper proposes an occluded expression recognition model based on the generated countermeasure network. The model is divided into two modules, namely, occluded face image restoration and face recognition. MethodsFirstly, this paper summarizes the research status of deep facial expression recognition methods in recent ten years and the development of related facial expression database. Then, the current facial expression recognition methods based on deep learning are divided into two categories: Static facial expression recognition and dynamic facial expression recognition. The two methodswill be introduced and summarized respectively. Aiming at the advanced deep expression recognition algorithms in the field, the performance of these algorithms on common expression databases is compared, and the strengths and weaknesses of these algorithms are analyzed in detail. Discussion and resultsAs the task of facial expression recognition is gradually transferred from the controlled laboratory environment to the challenging real-world environment, with the rapid development of deep learning technology, deep neural network can learn discriminative features, and is gradually applied to automatic facial expression recognition task. The current deep facial expression recognition system is committed to solve the following two problems: (1) Overfitting due to lack of sufficient training data; (2) In the real world environment, other variables that have nothing to do with expression bring interference problems. ConclusionFrom the perspective of algorithm, combining other expression models, such as facial action unit model and pleasure arousal dimension model, as well as other multimodal models, such as audio mode, 3D face depth information and human physiological information, can make expression recognition more practical.

Dynamic facial expression recognition of sprinters based on multi-scale detail enhancement

Dynamic facial expression recognition of sprinters based on multi-scale detail enhancement

Dynamic facial expression recognition of sprinters based on multi-scale detail enhancement

In order to solve the problems of low average gradient and long recognition time in traditional facial expression recognition method, a multi-scale detail enhancement method for facial dynamic expression recognition of sprint athletes is proposed. A principal component analysis method was used to establish the facial expression feature subspace of sprinters, to project and reduce the dimension of the facial dynamic expression feature vector of sprinters, and to obtain the low frequency information and high frequency information of the facial image of sprinters by bilateral filtering. The multi-scale details of expression are enhanced by using side suppression network model and improving image S curve. The feature vector of facial dynamic expression is input into support vector machine to recognise the facial dynamic expression of sprinter. Experimental results show that the average value of annoying gradient is about 98 and the shortest time s. is about 1.9.