Facial Expression Recognition Task Research Articles

A joint local spatial and global temporal CNN-Transformer for dynamic facial expression recognition

Unlike conventional video action recognition, Dynamic Facial Expression Recognition (DFER) tasks exhibit minimal spatial movement of objects. Addressing this distinctive attribute, we propose an innovative CNN-Transformer model, named LSGTNet, specifically tailored for DFER tasks. Our LSGTNet comprises three stages, each composed of a spatial CNN (Spa-CNN) and a temporal transformer (T-Former) in sequential order. The Spa-CNN extracts spatial features from images, yielding smaller-sized feature maps to alleviate the computational complexity for subsequent T-Former. The T-Former integrates global temporal information from the same spatial positions across different time frames while retaining the feature map dimensions. The alternating interplay between Spa-CNN and T-Former ensures a continuous fusion of spatial and temporal information, leading our model to excel across various real-world datasets. To the best of our knowledge, this is the first method to address the DFER challenge by focusing on capturing the temporal changes in muscles within local spatial regions. Our method has achieved state-of-the-art results on multiple in-the-wild datasets and datasets under laboratory conditions.

Synchrony or asynchrony: development of facial expression recognition from childhood to adolescence based on large-scale evidence.

The development of facial expression recognition ability in children is crucial for their emotional cognition and social interactions. In this study, 510 children aged between 6 and 15 participated in a two forced-choice task of facial expression recognition. The findings supported that recognition of the six basic facial expressions reached a relatively stable mature level around 8-9 years old. Additionally, model fitting results indicated that children showed the most significant improvement in recognizing expressions of disgust, closely followed by fear. Conversely, recognition of expressions of happiness and sadness showed slower improvement across different age groups. Regarding gender differences, girls exhibited a more pronounced advantage. Further model fitting revealed that boys showed more pronounced improvements in recognizing expressions of disgust, fear, and anger, while girls showed more pronounced improvements in recognizing expressions of surprise, sadness, and happiness. These clear findings suggested the synchronous developmental trajectory of facial expression recognition from childhood to adolescence, likely influenced by socialization processes and interactions related to brain maturation.

Facial expression recognition ability and its neuropsychological mechanisms in children with attention deficit and hyperactive disorder.

Attention deficit and hyperactive disorder (ADHD) is a chronic neurodevelopmental disorder characterized by inattention, hyperactivity-impulsivity, and working memory deficits. Social dysfunction is one of the major challenges faced by children with ADHD. It has been found that children with ADHD can't perform as well as typically developing children on facial expression recognition (FER) tasks. Generally, children with ADHD have some difficulties in FER, while some studies suggest that they have no significant differences in accuracy of specific emotion recognition compared with typically developing children. The neuropsychological mechanisms underlying these difficulties are as follows. First, neuroanatomically. Compared to typically developing children, children with ADHD show smaller gray matter volume and surface area in the amygdala and medial prefrontal cortex regions, as well as reduced density and volume of axons/cells in certain frontal white matter fiber tracts. Second, neurophysiologically. Children with ADHD exhibit increased slow-wave activity in their electroencephalogram, and event-related potential studies reveal abnormalities in emotional regulation and responses to angry faces when facing facial stimuli. Third, psychologically. Psychosocial stressors may influence FER abilities in children with ADHD, and sleep deprivation in ADHD children may significantly increase their recognition threshold for negative expressions such as sadness and anger. This article reviews research progress over the past three years on FER abilities of children with ADHD, analyzing the FER deficit in children with ADHD from three dimensions: neuroanatomy, neurophysiology and psychology, aiming to provide new perspectives for further research and clinical treatment of ADHD.

Automatic facial expression recognition under partial occlusion based on motion reconstruction using a denoising autoencoder

Automatic facial expression recognition (FER) plays a valuable role in various fields, including health, road safety, and marketing, where providing feedback on the user’s condition is crucial. While significant progress has been made in controlled environments (such as frontal, unconcluded, and well-lit conditions), recognizing facial expressions in unconstrained environments (natural settings) remains challenging. The presence of occlusions poses a particular difficulty as they obscure parts of the facial information captured in the image. To address this issue, researchers have proposed different solutions, broadly categorized into two approaches: those focusing on visible regions of the face and those attempting to reconstruct hidden parts. Currently, most solutions rely on texture or geometry-based methods, with only a few utilizing motion-based approaches. However, incorporating motion appears to be particularly promising in adapting to occlusions due to its unique characteristics, such as close-range propagation and local coherence. In this paper, our focus lies on leveraging motion to overcome the challenges posed by occlusions in FER tasks.

Common Latent Embedding Space for Cross-Domain Facial Expression Recognition

In practical facial expression recognition (FER), the training data and test data are often obtained from different domains. It is obvious that the domain disparity could significantly degrade the recognition performance. To tackle this challenging cross-domain FER problem, we put forward a novel method termed common latent embedding space (CLES). To be specific, first, we obtain a common embedding space for cross-domain samples by matrix factorization (MF). Then, the dual-graph Laplacian is applied to this common embedding space to narrow the gap across distinct domains and, meanwhile, explores the inherent geometric information. Furthermore, to characterize the global relationship of the cross-domain samples, the self-representation strategy is used to guide the learning of the common embedding space. Finally, comprehensive experiments on four benchmark databases indicate that the proposed method can achieve better performance in comparison with the state-of-the-art methods on cross-domain FER tasks.

Open-Set Facial Expression Recognition

Facial expression recognition (FER) models are typically trained on datasets with a fixed number of seven basic classes. However, recent research works (Cowen et al. 2021; Bryant et al. 2022; Kollias 2023) point out that there are far more expressions than the basic ones. Thus, when these models are deployed in the real world, they may encounter unknown classes, such as compound expressions that cannot be classified into existing basic classes. To address this issue, we propose the open-set FER task for the first time. Though there are many existing open-set recognition methods, we argue that they do not work well for open-set FER because FER data are all human faces with very small inter-class distances, which makes the open-set samples very similar to close-set samples. In this paper, we are the first to transform the disadvantage of small inter-class distance into an advantage by proposing a new way for open-set FER. Specifically, we find that small inter-class distance allows for sparsely distributed pseudo labels of open-set samples, which can be viewed as symmetric noisy labels. Based on this novel observation, we convert the open-set FER to a noisy label detection problem. We further propose a novel method that incorporates attention map consistency and cycle training to detect the open-set samples. Extensive experiments on various FER datasets demonstrate that our method clearly outperforms state-of-the-art open-set recognition methods by large margins. Code is available at https://github.com/zyh-uaiaaaa.

Effective Facial Expression Recognition System Using Machine Learning

The co Facial expression recognition (FER) is a topic that has seen a lot of study in computer vision and machine learning. In recent years, deep learning techniques have shown remarkable progress on FER tasks. With this abstract, A Novel Is Advised By Us FER method that combines combined use of k-nearest neighbours and long short-term memory algorithms better efficiency and accurate facial expression recognition. The proposed system features two primary steps—feature extraction and classification—to get results. When extracting features, we extract features from the facial images using the Local Binary Patterns (LBP) algorithm. LBP is a simple yet powerful feature extraction technique that captures texture information from the image. In the classification stage, we use the KNN and LSTM algorithms for facial expression recognition. KNN is a simple and effective classification algorithm that finds the k closest to the given value neighbours to the test training-set-sample and assigning it to the class that is most frequent among its neighbours. However, KNN has limitations in handling temporal information. To address this limitation, we propose to use LSTM, which is a subclass of RNNs capable of capturing temporal relationships in time series data. The LSTM network takes as input the LBP features of a sequence of facial images and processes them through a series of LSTM cells to estimate the ultimate coding of the phrase. We examine the planned and system on two publicly available records: the CK+ and the Oulu-CASIA datasets. According on the experimental findings, the proposed system achieves performance at the cutting edge on both datasets. The proposed system performs better than other state-of-the-art methods, including those that use deep learning systems, quantitatively, in terms of F1-score and precision.In conclusion, the proposed FER system that combines KNN and LSTM algorithms achieves high accuracy and an F1 score in recognising facial expressions from sequences of images. This system can be used in many contexts, including human-computer interaction, emotion detection, and behaviour analysis.

BPMB: BayesCNNs with perturbed multi-branch structure for robust facial expression recognition

Wild Facial Expression Recognition (FER) task has been a long-standing challenge due to the various forms of uncertainty exist in expression data. When expression data is fed into a convolutional neural network (CNN), the model's estimated parameters also become uncertain. This uncertainty gives rise to concerns regarding the reliability of the recognition results. To quantify these uncertainties and achieve robust performance in the presence of noisy data, this paper introduces a novel model for Wild Facial Expression Recognition: the Bayesian Convolutional Neural Network with Perturbed Multi-Branch Structure (BPMB). This model aims to address uncertainty issues, enabling the network's decisions to become more deterministic with increasing training accuracy. Specifically, BPMB incorporates variational inference (VI) to introduce a probability distribution for the weights. A variational approximation to the true posterior is derived using Bayes by Backprop, involving two convolution operations: one for classification and another for quantifying uncertainty. Furthermore, an exploration is conducted into a lightweight multi-branch structure that leverages Dropout as a random generator to introduce perturbations during the training process, enhancing the model's robustness while extracting deep features. Extensive experiments validate the superiority of the proposed BPMB algorithm over the majority of existing mainstream algorithms on three widely utilized wild datasets.

CF-DAN: Facial-expression recognition based on cross-fusion dual-attention network

Recently, facial-expression recognition (FER) has primarily focused on images in the wild, including factors such as face occlusion and image blurring, rather than laboratory images. Complex field environments have introduced new challenges to FER. To address these challenges, this study proposes a cross-fusion dual-attention network. The network comprises three parts: (1) a cross-fusion grouped dual-attention mechanism to refine local features and obtain global information; (2) a proposed C2 activation function construction method, which is a piecewise cubic polynomial with three degrees of freedom, requiring less computation with improved flexibility and recognition abilities, which can better address slow running speeds and neuron inactivation problems; and (3) a closed-loop operation between the self-attention distillation process and residual connections to suppress redundant information and improve the generalization ability of the model. The recognition accuracies on the RAF-DB, FERPlus, and AffectNet datasets were 92.78%, 92.02%, and 63.58%, respectively. Experiments show that this model can provide more effective solutions for FER tasks.

The effect of age on emotion processing in individuals with mood disorders and in healthy individuals.

Emotion processing is an essential part of interpersonal relationships and social interactions. Changes in emotion processing have been found in both mood disorders and in aging, however, the interaction between such factors has yet to be examined in detail. This is of interest due to the contrary nature of the changes observed in existing research - a negativity bias in mood disorders versus a positivity effect with aging. It is also unclear how changes in non-emotional cognitive function with aging and in mood disorders, interact with these biases. In individuals with mood disorders and in healthy control participants, we examined emotional processing and its relationship to age in detail. Data sets from two studies examining facial expression recognition were pooled. In one study, 98 currently depressed individuals (either unipolar or bipolar) were compared with 61 healthy control participants, and in the other, 100 people with bipolar disorder (in various mood states) were tested on the same facial expression recognition task. Repeated measures analysis of variance was used to examine the effects of age and mood disorder diagnosis alongside interactions between individual emotion, age, and mood disorder diagnosis. A positivity effect was associated with increasing age which was evident irrespective of the presence of mood disorder or current mood episode. Results suggest a positivity effect occurring at a relatively early age but with no evidence of a bias toward negative emotions in mood disorder or specifically, in depressed episodes. The positivity effect in emotional processing in aging appears to occur even within people with mood disorders. Further research is needed to understand how this fits with negative biases seen in previous studies in mood disorders.

Cross-Layer Contrastive Learning of Latent Semantics for Facial Expression Recognition.

Convolutional neural networks (CNNs) have achieved significant improvement for the task of facial expression recognition. However, current training still suffers from the inconsistent learning intensities among different layers, i.e., the feature representations in the shallow layers are not sufficiently learned compared with those in deep layers. To this end, this work proposes a contrastive learning framework to align the feature semantics of shallow and deep layers, followed by an attention module for representing the multi-scale features in the weight-adaptive manner. The proposed algorithm has three main merits. First, the learning intensity, defined as the magnitude of the backpropagation gradient, of the features on the shallow layer is enhanced by cross-layer contrastive learning. Second, the latent semantics in the shallow-layer and deep-layer features are explored and aligned in the contrastive learning, and thus the fine-grained characteristics of expressions can be taken into account for the feature representation learning. Third, by integrating the multi-scale features from multiple layers with an attention module, our algorithm achieved the state-of-the-art performances, i.e. 92.21%, 89.50%, 62.82%, on three in-the-wild expression databases, i.e. RAF-DB, FERPlus, SFEW, and the second best performance, i.e. 65.29% on AffectNet dataset. Our codes will be made publicly available.

Empower smart cities with sampling-wise dynamic facial expression recognition via frame-sequence contrastive learning

In the construction of smart cities, facial expression analysis plays a crucial role. It can be used in traffic monitoring systems to alleviate traffic pressure by analyzing the emotional states of drivers and passengers. In the field of smart healthcare, it can provide more precise treatment and services to patients. In the realm of social entertainment, it can offer more intelligent and personalized interactions. In summary, the application of emotion computing technology will play an increasingly significant role in the development of smart cities in the future. In the task of dynamic facial expression recognition (DFER), analyzing the spatial–temporal features of video sequences has become a common research approach. However, facial expression sequences often contain a significant number of neutral frames and noisy frames, potentially increasing computational costs and reducing performance. Effectively extracting key frames for spatial–temporal feature analysis is a critical aspect of dynamic facial expression recognition. To address this issue, we proposed a sampling-wise dynamic facial expression recognition via frame-Sequence contrastive learning method, called SW-FSCL. The SW-FSCL method aims to improve the performance of DFER by using intelligent dual-stream sampling strategies and frame-sequence contrastive learning, extract key frame and reduce the impact of neutral frames and noisy frames. We proposed a key frame proposal (KFP) block to analyze the spatial–temporal features of sequences, calculating weight ratios for key frame extraction. Due to potential information loss in long sequences, we introduce a temporal aggregation (TA) block to prevent data loss and ensure the integrity of temporal information. The experimental results provide compelling evidence that the proposed approach not only outperforms all current state-of-the-art algorithms on two widely-used benchmark datasets (DFEW, FERV39k), but also visualization results produces insights into the interpretability of the SW-FSCL method.

A joint learning method with consistency-aware for low-resolution facial expression recognition

Existing facial expression recognition (FER) methods are mainly devoted to learning discriminative features from high-resolution images. However, when applied to low-resolution images, their performance drops rapidly. This paper proposes a unified learning framework (namely SR-FER) by cascading the image super-resolution (SR) task and FER task to alleviate the low-resolution challenge. It effectively feeds back expression-related information from the FER network to the SR network, and returns the quality-enhanced expression images via a SR network. Specifically, a multi-stage attention-aware consistency loss module is introduced to help the SR network achieve discriminative feature restoration guided by attention information. Furthermore, a prediction consistency loss module is also developed to encourage the SR network to restore discriminative features by reducing the difference in prediction information between the restored and original normal-resolution images. Therefore, more accurate results are obtained by performing FER on the restored images. We conduct extensive experiments to demonstrate that the proposed low-resolution FER solution can help SR methods restore features favorable for FER while maintaining acceptable FER performance in various resolution degradation scenarios. The proposed method effectively improves the FER challenge under resolution degradation conditions, which is of good reference value for real-world applications.

A multi‐scale feature fusion convolutional neural network for facial expression recognition

AbstractThis paper designs a new facial Expression recognition network called a multi‐scale feature Fusion Convolutional neural Network (EFCN). This network is proposed to solve two problems in the facial expression recognition task. First, there are many commonalities between faces of different expression categories, and the recognition task cannot be precisely performed when the commonality is greater than the individuality. Secondly, facial detail features have a significant impact on the final results of expression recognition, while the image detail features extracted by traditional convolutional neural networks are not sufficient. In order to address the above issues, the feature enhancement network (FEN) and the detail information enhancement module (DEM) are designed. The FEN fuses deep and shallow features. Accordingly, the feature map contains richer information, making it easy to identify the samples. The DEM extracts and fuses the features passed by the backbone network with multi‐scale features to enhance the network's ability to extract features from small regions of the face. We validated the proposed method on three datasets, RAF‐DB, CK+, and JAFFE, and achieved 84.50%, 97.86%, and 91.05% accuracy, respectively, and the experimental results showed the effectiveness of the proposed method in this paper. For example, on the JAFFE dataset, the recognition accuracy of this method surpasses the MLT method by 1.87%.

Joint recognition of basic and compound facial expressions by mining latent soft labels

Previous works on facial expression recognition focus on basic emotions, while ignoring more complex compound expressions. However, both basic and compound emotions appear in the real-world environment. In this work, we aim to jointly recognize basic and compound expressions. Aiming at the Basic-Compound Facial Expression Recognition (BC-FER) task, we illustrate that traditional hard label training is not ideal due to great label dependencies. Therefore, we propose an expression soft label mining (ESLM) method to improve the performance. On the one hand, an iterated soft label mining (ISLM) algorithm assisted by teacher–student network is proposed to make the network generate soft targets automatically for learning. On the other hand, to explicitly leverage prior knowledge of label correlations, we propose an expression correlation score learning (ECSL) loss to regularize the predicted distributions. Extensive experimental results on CFEE, RAF-DB, and EmotioNet show that our method achieves state-of-the-art performance on BC-FER task.

Dual subspace manifold learning based on GCN for intensity-invariant facial expression recognition

Facial expression recognition (FER) is one of the most important computer vision tasks for understanding human inner emotions. However, the poor generation ability of the FER model limits its applicability due to tremendous intraclass variation. Especially for expressions of varying intensities, the appearance differences among weak expressions are subtle, which makes FER tasks challenging. In response to these issues, this paper presents a dual subspace manifold learning method based on a graph convolutional network (GCN) for intensity-invariant FER tasks. Our method treats the target task as a node classification problem and learns the manifold representation using two subspace analysis methods: locality preserving projection (LPP) and peak-piloted locality preserving projection (PLPP). Inspired by the classic LPP, which maintains local similarity among data, this paper introduces a novel PLPP that maintains the locality between peak expressions and non-peak expressions to enhance the representation of weak expressions. This paper also reports two subspace fusion methods, one based on a weighted adjacency matrix and another on a self-attention mechanism, that combine the LPP and PLPP to further improve FER performance. The second method achieves a recognition accuracy of 93.83% on the CK+, 74.86% on the Oulu-CASIA and 75.37% on the MMI for weak expressions, outperforming state-of-the-art methods.

Associative learning and facial expression recognition in schizophrenic patients: Effects of social presence

Schizophrenia (SCZ) is a psychiatric disorder that alters both general and social cognition. However, the exact mechanisms that are altered remain to be elucidated. In this study, we investigated associative learning (AL) and facial expression recognition (FER) in the same patients, using emotional expressions and abstract images. Our main aim was to investigate how these cognitive abilities are affected by SCZ and to assess the role of mere social presence, a factor that has not been considered before. The study compared the behavioral performance of 60 treated outpatients with SCZ and 103 demographically matched healthy volunteers. In the AL task, participants had to associate abstract images or facial expressions with key presses, guided by feedback on each trial. In the FER task, they had to report whether two successively presented facial expressions were the same or different. All participants performed the two tasks under two social context conditions: alone or in the presence of a passive but attentive audience. The results showed a severe learning impairment in patients compared to controls, with a slight advantage for facial expressions compared to abstract images, and a gender-dependent effect of social presence. In contrast, facial expression recognition was partially spared in patients and facilitated by social presence. We conclude that cognitive abilities are impaired in patients with SCZ, but their investigation needs to take into account the social context in which they are assessed.

Effect of the deep residual networks at different depths on expression recognition

Facial expression recognition plays a critical role in numerous applications like emotion analysis, human-computer interaction, and surveillance systems. Given the importance of this task, this study aims to investigate the effectiveness of different depths of Residual Networks (ResNet). The primary objective is to scrutinize and compare these ResNet models in terms of their training and validation losses and performance metrics like accuracy, recall, and F1 scores. In this research, a thorough comparative analysis is conducted by setting up exhaustive experiments using these models. The experiment is carried out on a popular facial expression dataset. Despite the depth differences, ResNet101 emerged as the model demonstrating superior performance. It struck the most effective balance between model complexity and generalization capacity, leading to the lowest validation loss and better performance. Experimental results show that a more complex model does not necessarily yield optimal results. The optimal balance between model complexity and generalisation needs to be investigated. These findings can provide essential guidance in the design of deep learning models for facial expression recognition and other similar tasks.

Bias-Based Soft Label Learning for Facial Expression Recognition

Facial Expression Recognition (FER) suffers from misrecognition due to the similarities between expressions. To address this issue, popular works replace original annotations with soft labels to reflect expression similarities. However, existing soft label learning (SLL) modules are independent of FER modules. In this paper, inspired by automatic control theory, we propose a bias-based soft label learning network for FER named EC-Net. For optimizing FER and SLL modules jointly, EC-Net constitutes the closed-loop feedback between the two modules by designing a module measuring and transmitting the bias between FER module predictions and target labels. Specifically, EC-Net contains three modules: E-subNet, C-subNet, and L-Transmitter. Firstly, E-subNet, i.e., the FER module, attempts to converge to target labels under the supervision of soft labels, acting as the executor. Then, L-Transmitter measures the bias between E-subNet predictions and target labels. It converts multiple discrete biases to the bias-based label through spectral clustering and transmits it to C-subNet. Finally, C-SubNet, i.e., the SLL module, generates soft labels from the bias-based label with a cascaded learner and progressively distinguishes similar expressions. It updates the learned soft labels for E-subNet, performing like the controller. Supervised by the bias-based soft label, E-subNet effectively reduces the dominant bias caused by similar expressions. We conduct extensive experiments on four popular benchmarks, demonstrating the effectiveness of applying closed-loop feedback in the FER task.

Pose-invariant and occlusion-robust neonatal facial pain assessment

Neonatal Facial Pain Assessment (NFPA) is essential to improve neonatal pain management. Pose variation and occlusion, which can significantly alter the facial appearance, are two major and still unstudied barriers to NFPA. We bridge this gap in terms of method and dataset. Techniques to tackle both challenges in other tasks either expect pose/occlusion-invariant deep learning methods or first generate a normal version of the input image before feature extraction, combining these we argue that it is more effective to jointly perform adversarial learning and end-to-end classification for their mutual benefit. To this end, we propose a Pose-invariant Occlusion-robust Pain Assessment (POPA) framework, with two novelties. We incorporate adversarial learning-based disturbance mitigation for end-to-end pain-level classification and propose a novel composite loss function for facial representation learning; compared to the vanilla discriminator that implicitly determines occlusion and pose conditions, we propose a multi-scale discriminator that determines explicitly, while incorporating local discriminators to enhance the discrimination of key regions. For a comprehensive evaluation, we built the first neonatal pain dataset with disturbance annotation involving 1091 neonates and also applied the proposed POPA to the facial expression recognition task. Extensive qualitative and quantitative experiments prove the superiority of the POPA.