Learning For Facial Expression Recognition Research Articles

ASDC-FER: attention-guided self-supervised distilled contrastive learning for facial expression recognition

ASDC-FER: attention-guided self-supervised distilled contrastive learning for facial expression recognition

Robust consistency learning for facial expression recognition under label noise

Robust consistency learning for facial expression recognition under label noise

Multi-threshold deep metric learning for facial expression recognition

Feature representations generated through triplet-based deep metric learning offer significant advantages for facial expression recognition (FER). Each threshold in triplet loss inherently shapes a distinct distribution of inter-class variations, leading to unique representations of expression features. Nonetheless, pinpointing the optimal threshold for triplet loss presents a formidable challenge, as the ideal threshold varies not only across different datasets but also among classes within the same dataset. In this paper, we propose a novel multi-threshold deep metric learning approach that bypasses the complex process of threshold validation and markedly improves the effectiveness in creating expression feature representations. Instead of choosing a single optimal threshold from a valid range, we comprehensively sample thresholds throughout this range, which ensures that the representation characteristics exhibited by the thresholds within this spectrum are fully captured and utilized for enhancing FER. Specifically, we segment the embedding layer of the deep metric learning network into multiple slices, with each slice representing a specific threshold sample. We subsequently train these embedding slices in an end-to-end fashion, applying triplet loss at its associated threshold to each slice, which results in a collection of unique expression features corresponding to each embedding slice. Moreover, we identify the issue that the traditional triplet loss may struggle to converge when employing the widely-used Batch Hard strategy for mining informative triplets, and introduce a novel loss termed dual triplet loss to address it. Extensive evaluations demonstrate the superior performance of the proposed approach on both posed and spontaneous facial expression datasets.

Multi-class differentiation feature representation guided joint dictionary learning for facial expression recognition

Multi-class differentiation feature representation guided joint dictionary learning for facial expression recognition

Combining ViT with contrastive learning for facial expression recognition

Combining ViT with contrastive learning for facial expression recognition

Attention-guided local feature joint learning for facial expression recognition

Attention-guided local feature joint learning for facial expression recognition

MMATrans: Muscle Movement Aware Representation Learning for Facial Expression Recognition via Transformers

MMATrans: Muscle Movement Aware Representation Learning for Facial Expression Recognition via Transformers

DR-FER: Discriminative and Robust Representation Learning for Facial Expression Recognition

DR-FER: Discriminative and Robust Representation Learning for Facial Expression Recognition

Bias-Based Soft Label Learning for Facial Expression Recognition

Facial Expression Recognition (FER) suffers from misrecognition due to the similarities between expressions. To address this issue, popular works replace original annotations with soft labels to reflect expression similarities. However, existing soft label learning (SLL) modules are independent of FER modules. In this paper, inspired by automatic control theory, we propose a bias-based soft label learning network for FER named EC-Net. For optimizing FER and SLL modules jointly, EC-Net constitutes the closed-loop feedback between the two modules by designing a module measuring and transmitting the bias between FER module predictions and target labels. Specifically, EC-Net contains three modules: E-subNet, C-subNet, and L-Transmitter. Firstly, E-subNet, i.e., the FER module, attempts to converge to target labels under the supervision of soft labels, acting as the executor. Then, L-Transmitter measures the bias between E-subNet predictions and target labels. It converts multiple discrete biases to the bias-based label through spectral clustering and transmits it to C-subNet. Finally, C-SubNet, i.e., the SLL module, generates soft labels from the bias-based label with a cascaded learner and progressively distinguishes similar expressions. It updates the learned soft labels for E-subNet, performing like the controller. Supervised by the bias-based soft label, E-subNet effectively reduces the dominant bias caused by similar expressions. We conduct extensive experiments on four popular benchmarks, demonstrating the effectiveness of applying closed-loop feedback in the FER task.

Expression-Guided Deep Joint Learning for Facial Expression Recognition.

In recent years, convolutional neural networks (CNNs) have played a dominant role in facial expression recognition. While CNN-based methods have achieved remarkable success, they are notorious for having an excessive number of parameters, and they rely on a large amount of manually annotated data. To address this challenge, we expand the number of training samples by learning expressions from a face recognition dataset to reduce the impact of a small number of samples on the network training. In the proposed deep joint learning framework, the deep features of the face recognition dataset are clustered, and simultaneously, the parameters of an efficient CNN are learned, thereby marking the data for network training automatically and efficiently. Specifically, first, we develop a new efficient CNN based on the proposed affinity convolution module with much lower computational overhead for deep feature learning and expression classification. Then, we develop an expression-guided deep facial clustering approach to cluster the deep features and generate abundant expression labels from the face recognition dataset. Finally, the AC-based CNN is fine-tuned using an updated training set and a combined loss function. Our framework is evaluated on several challenging facial expression recognition datasets as well as a self-collected dataset. In the context of facial expression recognition applied to the field of education, our proposed method achieved an impressive accuracy of 95.87% on the self-collected dataset, surpassing other existing methods.

Combining deep subspace feature representation based IKPCANet and jointly constraint multi-dictionary learning for facial expression recognition

Combining deep subspace feature representation based IKPCANet and jointly constraint multi-dictionary learning for facial expression recognition

Few-shot learning for facial expression recognition: a comprehensive survey

Facial expression recognition (FER) is utilized in various fields that analyze facial expressions. FER is attracting increasing attention for its role in improving the convenience in human life. It is widely applied in human–computer interaction tasks. However, recently, FER tasks have encountered certain data and training issues. To address these issues in FER, few-shot learning (FSL) has been researched as a new approach. In this paper, we focus on analyzing FER techniques based on FSL and consider the computational complexity and processing time in these models. FSL has been researched as it can solve the problems of training with few datasets and generalizing in a wild-environmental condition. Based on our analysis, we describe certain existing challenges in the use of FSL in FER systems and suggest research directions to resolve these issues. FER using FSL can be time efficient and reduce the complexity in many other real-time processing tasks and is an important area for further research.

Self-supervised vision transformer-based few-shot learning for facial expression recognition

Facial expression recognition (FER) is embedded in many real-world human-computer interaction tasks, such as online learning, depression recognition and remote diagnosis. However, FER is often hindered by privacy concerns and low recognition accuracy due to inadequate data transfer restrictions on public clouds, insufficient quantities of effective labeled samples and class imbalance. To address the above challenges, we have developed an automatic privacy-preserving learning state recognition system for supervising the quality of online teaching with the cooperation of edge servers and cloud servers to reduce the risk of privacy exposure. In particular, we propose few-shot facial expression recognition with a self-supervised vision transformer (SSF-ViT) by integrating self-supervised learning (SSL) and few-shot learning (FSL) to train a deep learning model with fewer labeled samples. Specifically, a vision transformer (ViT) is jointly pretrained with four self-supervised pretext tasks, including image denoising and reconstruction, image rotation prediction, jigsaw puzzle and masked patch prediction, to obtain a pretrained ViT encoder. Then, the pretrained ViT encoder is used on a lab-controlled labeled FER dataset to extract the spatiotemporal features and implement the FER task to fine-tune the parameters. Finally, we construct prototypes to verify the few-shot classification method for specific expression recognition. Support and query sets are divided in the wild FER dataset, and few-shot classification episodes are constructed. The fine-tuned ViT encoder is used as the feature extractor to build the prototype for each support set category, and the expression classification results are obtained by computing the Euclidean distance between the query samples and the prototypes. The extensive experimental results show that SSF-ViT can achieve recognition accuracies of 74.95%, 66.04%, 63.69% and 90.98% on the FER2013, AffectNet, SFEW 2.0 and RAF-DB datasets, respectively. In addition, SSF-ViT can improve the recognition performance of specific expression categories on these datasets.

A Brief Review of Deep Learning for Facial Expression Recognition

A Brief Review of Deep Learning for Facial Expression Recognition

SSA-ICL: Multi-domain adaptive attention with intra-dataset continual learning for Facial expression recognition

Facial expression recognition (FER) is a kind of affective computing that identifies the emotional state represented in facial photographs. Various methods have been developed for completing this critical task. In spite of this progress, three significant obstacles, the interaction between spatial action units, the inadequacy of semantic information about spectral expressions and the unbalanced data distribution, are not well addressed. In this work, we propose SSA-ICL, a novel approach for FER, and solve these three difficulties inside a coherent framework. To address the first two challenges, we develop a Spectral and Spatial Attention (SSA) module that integrates spectral semantics with spatial locations to improve the performance of the model. We provide an Intra-dataset Continual Learning (ICL) module to combat the issue of long-tail distribution in FER datasets. By subdividing a single long-tail dataset into multiple sub-datasets, ICL repeatedly trains well-balanced representations from each subset and finally develop a independent classifier. We performed extensive experiments on two publicly available datasets, AffectNet and RAFDB. In comparison to existing attention modules, our SSA achieves an accuracy improvement of 3.8%∼6.7%, as evidenced by testing results. In the meanwhile, our proposed SSA-ICL can achieve superior or comparable performance to state-of-the-art FER methods (65.78% on AffectNet and 89.44% on RAFDB).

Multi-stage and multi-branch network with similar expressions label distribution learning for facial expression recognition

This paper proposes a novel facial expression recognition network, called Multi-Stage and Similar Expressions Label Distribution Learning Network (MSCL). Our method is based on the observations of the labels ambiguity between similar expressions in complex wild scenes, for there are inherently similar features between them that are difficult to distinguish and even manually mislabeled. The proposed network consists of three modules, namely a multi-stage multi-branch classification network (MSB), a multi-branch label distribution learning module (MLD), and a multi-branch similarity preserving module (MSP). MSB aggregates similar expression features through the first-stage prediction, MLD utilizes the aggregation results of MSB to extract the label distribution between similar expressions, and MSP utilizes a consistency relationship to minimize the differences between multiple branches. We propose an end-to-end model, where each module can be integrated with existing network modules. Furthermore, our method achieves 89.44% accuracy on the RAF-DB dataset and achieves state-of-the-art results on the AffectNet dataset with 63.25% and 66.56% accuracy on its two subsets, AffectNet-8 and AffectNet-7, respectively.

Hierarchical Spatial Pyramidal Feature Representation Learning for Facial Expression Recognition

As the recognition of facial expressions has attracted increasing attention due to its important applications in several fields (security, human-machine interaction, marketing, etc.), thanks to the detection of different possible expressions (happiness, sadness, etc.), this technology is gaining more attention. This article presents a method for the recognition of facial expressions by exploiting spatial information. In this method, the LBP descriptor is proposed to extract local characteristics, and a hierarchical spatial pyramid model is used to capture spatial information in an image. This method enriches the semantic description of visual information, which improves the recognition rate of facial expressions. The experimental analysis was carried out on the CK+ facial expression dataset, which includes seven categories of expressions—“happy,” “sad,” “surprise,” “contempt,” “fear,” “disgust,” “anger”—and it has been given state-of-the-art performance. The recognition results of the system are very satisfactory.

HistNet: Histogram-based convolutional neural network with Chi-squared deep metric learning for facial expression recognition

Facial expression recognition is a challenging problem in machine learning. There is much research has been conducted in this field; however, the accuracy of facial expression recognition, especially in uncontrolled conditions, needs much improvement. In this paper, a deep histogram metric learning in a Convolutional Neural Network (CNN) is presented for facial expression recognition. The proposed CNN utilizes a histogram calculation layer to provide statistical description of feature maps at the output of the convolutional layers. To train the proposed CNN in histogram space, a learnable matrix (equivalent to the fully connected layer) is introduced in chi-squared distance equation. Then, the modified equation is used in the loss function. The recognition rates of the proposed CNN for seven-class facial expression recognition on four well-known databases including CK+, MMI, SFEW, and RAF-DB are 98.47%, 83.41%, 61.01%, and 89.28%, respectively. The results show superiority of the proposed CNN compared to the state-of-the-art methods.

Weighted ensemble with angular feature learning for facial expression recognition

Ensemble learning using a set of deep convolutional neural networks (DCNNs) as weak classifiers has become a powerful tool for face expression. Nevertheless, training a DCNNS-based ensemble is not only time consuming but also gives rise to high redundancy due to the nature of DCNNs. In this paper, a novel DCNNs-based ensemble method, named weighted ensemble with angular feature learning (WDEA), is proposed to improve the computational efficiency and diversity of the ensemble. Specifically, the proposed ensemble consists of four parts including input layer, trunk layers, diversity layers and loss fusion. Among them, the trunk layers which are used to extract the local features of face images are shared by diversity layers such that the lower-level redundancy can be largely reduced. The independent branches enable the diversity of the ensemble. Rather than the traditional softmax loss, the angular softmax loss is employed to extract more discriminant deep feature representation. Moreover, a novel weighting technique is proposed to enhance the diversity of the ensemble. Extensive experiments were performed on CK+ and AffectNet. Experimental results demonstrate that the proposed WDEA outperforms existing ensemble learning methods on the recogntion rate and computational efficiency.

Quantum-inspired meta-heuristic algorithms with deep learning for facial expression recognition under varying yaw angles

In recent years, the increasing human–computer interaction has spurred the interest of researchers towards facial expression recognition to determine the expressive changes in human beings. The detection of relevant features that describe the expressions of different individuals is vital to describe human expressions accurately. The present work has employed the integrated concept of Local Binary Pattern and Histogram of Gradient for facial feature extraction. The major contribution of the paper is the optimization of the extracted features using quantum-inspired meta-heuristic algorithms of QGA (Quantum-Inspired Genetic Algorithm), QGSA (Quantum-Inspired Gravitational Search Algorithm), QPSO (Quantum-Inspired Particle Swarm Optimization), and QFA (Quantum-Inspired Firefly Algorithm). These quantum-inspired meta-heuristic algorithms utilize the attributes of quantum computing that ensure the adequate control of facial feature diversity with quantum measures and Q-bit superstition states. The optimized features are fed to the deep learning (DL) variant deep convolutional neural network added with residual blocks (DCNN-R) for the classification of expressions. The facial expressions are detected for the KDEF and RaFD datasets under varying yaw angles of –90∘, –45∘, 0∘, 45∘, and 90∘. The detection of facial expressions with varying angles is also a crucial contribution, as the features decrease with the increasing yaw angle movement of the face. The experimental evaluations demonstrate the superior performance of the QFA than other algorithms for feature optimization and hence the better classification of facial expressions.