Learning For Facial Expression Recognition Research Articles

Multi-class differentiation feature representation guided joint dictionary learning for facial expression recognition

Multi-class differentiation feature representation guided joint dictionary learning for facial expression recognition

Combining ViT with contrastive learning for facial expression recognition

Combining ViT with contrastive learning for facial expression recognition

DR-FER: Discriminative and Robust Representation Learning for Facial Expression Recognition

DR-FER: Discriminative and Robust Representation Learning for Facial Expression Recognition

Bias-Based Soft Label Learning for Facial Expression Recognition

Facial Expression Recognition (FER) suffers from misrecognition due to the similarities between expressions. To address this issue, popular works replace original annotations with soft labels to reflect expression similarities. However, existing soft label learning (SLL) modules are independent of FER modules. In this paper, inspired by automatic control theory, we propose a bias-based soft label learning network for FER named EC-Net. For optimizing FER and SLL modules jointly, EC-Net constitutes the closed-loop feedback between the two modules by designing a module measuring and transmitting the bias between FER module predictions and target labels. Specifically, EC-Net contains three modules: E-subNet, C-subNet, and L-Transmitter. Firstly, E-subNet, i.e., the FER module, attempts to converge to target labels under the supervision of soft labels, acting as the executor. Then, L-Transmitter measures the bias between E-subNet predictions and target labels. It converts multiple discrete biases to the bias-based label through spectral clustering and transmits it to C-subNet. Finally, C-SubNet, i.e., the SLL module, generates soft labels from the bias-based label with a cascaded learner and progressively distinguishes similar expressions. It updates the learned soft labels for E-subNet, performing like the controller. Supervised by the bias-based soft label, E-subNet effectively reduces the dominant bias caused by similar expressions. We conduct extensive experiments on four popular benchmarks, demonstrating the effectiveness of applying closed-loop feedback in the FER task.

Expression-Guided Deep Joint Learning for Facial Expression Recognition.

In recent years, convolutional neural networks (CNNs) have played a dominant role in facial expression recognition. While CNN-based methods have achieved remarkable success, they are notorious for having an excessive number of parameters, and they rely on a large amount of manually annotated data. To address this challenge, we expand the number of training samples by learning expressions from a face recognition dataset to reduce the impact of a small number of samples on the network training. In the proposed deep joint learning framework, the deep features of the face recognition dataset are clustered, and simultaneously, the parameters of an efficient CNN are learned, thereby marking the data for network training automatically and efficiently. Specifically, first, we develop a new efficient CNN based on the proposed affinity convolution module with much lower computational overhead for deep feature learning and expression classification. Then, we develop an expression-guided deep facial clustering approach to cluster the deep features and generate abundant expression labels from the face recognition dataset. Finally, the AC-based CNN is fine-tuned using an updated training set and a combined loss function. Our framework is evaluated on several challenging facial expression recognition datasets as well as a self-collected dataset. In the context of facial expression recognition applied to the field of education, our proposed method achieved an impressive accuracy of 95.87% on the self-collected dataset, surpassing other existing methods.

Combining deep subspace feature representation based IKPCANet and jointly constraint multi-dictionary learning for facial expression recognition

Combining deep subspace feature representation based IKPCANet and jointly constraint multi-dictionary learning for facial expression recognition

Few-shot learning for facial expression recognition: a comprehensive survey

Facial expression recognition (FER) is utilized in various fields that analyze facial expressions. FER is attracting increasing attention for its role in improving the convenience in human life. It is widely applied in human–computer interaction tasks. However, recently, FER tasks have encountered certain data and training issues. To address these issues in FER, few-shot learning (FSL) has been researched as a new approach. In this paper, we focus on analyzing FER techniques based on FSL and consider the computational complexity and processing time in these models. FSL has been researched as it can solve the problems of training with few datasets and generalizing in a wild-environmental condition. Based on our analysis, we describe certain existing challenges in the use of FSL in FER systems and suggest research directions to resolve these issues. FER using FSL can be time efficient and reduce the complexity in many other real-time processing tasks and is an important area for further research.

Self-supervised vision transformer-based few-shot learning for facial expression recognition

Facial expression recognition (FER) is embedded in many real-world human-computer interaction tasks, such as online learning, depression recognition and remote diagnosis. However, FER is often hindered by privacy concerns and low recognition accuracy due to inadequate data transfer restrictions on public clouds, insufficient quantities of effective labeled samples and class imbalance. To address the above challenges, we have developed an automatic privacy-preserving learning state recognition system for supervising the quality of online teaching with the cooperation of edge servers and cloud servers to reduce the risk of privacy exposure. In particular, we propose few-shot facial expression recognition with a self-supervised vision transformer (SSF-ViT) by integrating self-supervised learning (SSL) and few-shot learning (FSL) to train a deep learning model with fewer labeled samples. Specifically, a vision transformer (ViT) is jointly pretrained with four self-supervised pretext tasks, including image denoising and reconstruction, image rotation prediction, jigsaw puzzle and masked patch prediction, to obtain a pretrained ViT encoder. Then, the pretrained ViT encoder is used on a lab-controlled labeled FER dataset to extract the spatiotemporal features and implement the FER task to fine-tune the parameters. Finally, we construct prototypes to verify the few-shot classification method for specific expression recognition. Support and query sets are divided in the wild FER dataset, and few-shot classification episodes are constructed. The fine-tuned ViT encoder is used as the feature extractor to build the prototype for each support set category, and the expression classification results are obtained by computing the Euclidean distance between the query samples and the prototypes. The extensive experimental results show that SSF-ViT can achieve recognition accuracies of 74.95%, 66.04%, 63.69% and 90.98% on the FER2013, AffectNet, SFEW 2.0 and RAF-DB datasets, respectively. In addition, SSF-ViT can improve the recognition performance of specific expression categories on these datasets.

A Brief Review of Deep Learning for Facial Expression Recognition

A Brief Review of Deep Learning for Facial Expression Recognition

SSA-ICL: Multi-domain adaptive attention with intra-dataset continual learning for Facial expression recognition

Facial expression recognition (FER) is a kind of affective computing that identifies the emotional state represented in facial photographs. Various methods have been developed for completing this critical task. In spite of this progress, three significant obstacles, the interaction between spatial action units, the inadequacy of semantic information about spectral expressions and the unbalanced data distribution, are not well addressed. In this work, we propose SSA-ICL, a novel approach for FER, and solve these three difficulties inside a coherent framework. To address the first two challenges, we develop a Spectral and Spatial Attention (SSA) module that integrates spectral semantics with spatial locations to improve the performance of the model. We provide an Intra-dataset Continual Learning (ICL) module to combat the issue of long-tail distribution in FER datasets. By subdividing a single long-tail dataset into multiple sub-datasets, ICL repeatedly trains well-balanced representations from each subset and finally develop a independent classifier. We performed extensive experiments on two publicly available datasets, AffectNet and RAFDB. In comparison to existing attention modules, our SSA achieves an accuracy improvement of 3.8%∼6.7%, as evidenced by testing results. In the meanwhile, our proposed SSA-ICL can achieve superior or comparable performance to state-of-the-art FER methods (65.78% on AffectNet and 89.44% on RAFDB).

Multi-stage and multi-branch network with similar expressions label distribution learning for facial expression recognition

This paper proposes a novel facial expression recognition network, called Multi-Stage and Similar Expressions Label Distribution Learning Network (MSCL). Our method is based on the observations of the labels ambiguity between similar expressions in complex wild scenes, for there are inherently similar features between them that are difficult to distinguish and even manually mislabeled. The proposed network consists of three modules, namely a multi-stage multi-branch classification network (MSB), a multi-branch label distribution learning module (MLD), and a multi-branch similarity preserving module (MSP). MSB aggregates similar expression features through the first-stage prediction, MLD utilizes the aggregation results of MSB to extract the label distribution between similar expressions, and MSP utilizes a consistency relationship to minimize the differences between multiple branches. We propose an end-to-end model, where each module can be integrated with existing network modules. Furthermore, our method achieves 89.44% accuracy on the RAF-DB dataset and achieves state-of-the-art results on the AffectNet dataset with 63.25% and 66.56% accuracy on its two subsets, AffectNet-8 and AffectNet-7, respectively.

Hierarchical Spatial Pyramidal Feature Representation Learning for Facial Expression Recognition

As the recognition of facial expressions has attracted increasing attention due to its important applications in several fields (security, human-machine interaction, marketing, etc.), thanks to the detection of different possible expressions (happiness, sadness, etc.), this technology is gaining more attention. This article presents a method for the recognition of facial expressions by exploiting spatial information. In this method, the LBP descriptor is proposed to extract local characteristics, and a hierarchical spatial pyramid model is used to capture spatial information in an image. This method enriches the semantic description of visual information, which improves the recognition rate of facial expressions. The experimental analysis was carried out on the CK+ facial expression dataset, which includes seven categories of expressions—“happy,” “sad,” “surprise,” “contempt,” “fear,” “disgust,” “anger”—and it has been given state-of-the-art performance. The recognition results of the system are very satisfactory.

HistNet: Histogram-based convolutional neural network with Chi-squared deep metric learning for facial expression recognition

Facial expression recognition is a challenging problem in machine learning. There is much research has been conducted in this field; however, the accuracy of facial expression recognition, especially in uncontrolled conditions, needs much improvement. In this paper, a deep histogram metric learning in a Convolutional Neural Network (CNN) is presented for facial expression recognition. The proposed CNN utilizes a histogram calculation layer to provide statistical description of feature maps at the output of the convolutional layers. To train the proposed CNN in histogram space, a learnable matrix (equivalent to the fully connected layer) is introduced in chi-squared distance equation. Then, the modified equation is used in the loss function. The recognition rates of the proposed CNN for seven-class facial expression recognition on four well-known databases including CK+, MMI, SFEW, and RAF-DB are 98.47%, 83.41%, 61.01%, and 89.28%, respectively. The results show superiority of the proposed CNN compared to the state-of-the-art methods.

Weighted ensemble with angular feature learning for facial expression recognition

Ensemble learning using a set of deep convolutional neural networks (DCNNs) as weak classifiers has become a powerful tool for face expression. Nevertheless, training a DCNNS-based ensemble is not only time consuming but also gives rise to high redundancy due to the nature of DCNNs. In this paper, a novel DCNNs-based ensemble method, named weighted ensemble with angular feature learning (WDEA), is proposed to improve the computational efficiency and diversity of the ensemble. Specifically, the proposed ensemble consists of four parts including input layer, trunk layers, diversity layers and loss fusion. Among them, the trunk layers which are used to extract the local features of face images are shared by diversity layers such that the lower-level redundancy can be largely reduced. The independent branches enable the diversity of the ensemble. Rather than the traditional softmax loss, the angular softmax loss is employed to extract more discriminant deep feature representation. Moreover, a novel weighting technique is proposed to enhance the diversity of the ensemble. Extensive experiments were performed on CK+ and AffectNet. Experimental results demonstrate that the proposed WDEA outperforms existing ensemble learning methods on the recogntion rate and computational efficiency.

Quantum-inspired meta-heuristic algorithms with deep learning for facial expression recognition under varying yaw angles

In recent years, the increasing human–computer interaction has spurred the interest of researchers towards facial expression recognition to determine the expressive changes in human beings. The detection of relevant features that describe the expressions of different individuals is vital to describe human expressions accurately. The present work has employed the integrated concept of Local Binary Pattern and Histogram of Gradient for facial feature extraction. The major contribution of the paper is the optimization of the extracted features using quantum-inspired meta-heuristic algorithms of QGA (Quantum-Inspired Genetic Algorithm), QGSA (Quantum-Inspired Gravitational Search Algorithm), QPSO (Quantum-Inspired Particle Swarm Optimization), and QFA (Quantum-Inspired Firefly Algorithm). These quantum-inspired meta-heuristic algorithms utilize the attributes of quantum computing that ensure the adequate control of facial feature diversity with quantum measures and Q-bit superstition states. The optimized features are fed to the deep learning (DL) variant deep convolutional neural network added with residual blocks (DCNN-R) for the classification of expressions. The facial expressions are detected for the KDEF and RaFD datasets under varying yaw angles of –90∘, –45∘, 0∘, 45∘, and 90∘. The detection of facial expressions with varying angles is also a crucial contribution, as the features decrease with the increasing yaw angle movement of the face. The experimental evaluations demonstrate the superior performance of the QFA than other algorithms for feature optimization and hence the better classification of facial expressions.

Discriminative attention-augmented feature learning for facial expression recognition in the wild

Facial expression recognition (FER) in-the-wild is challenging due to unconstraint settings such as varying head poses, illumination, and occlusions. In addition, the performance of a FER system significantly degrades due to large intra-class variation and inter-class similarity of facial expressions in real-world scenarios. To mitigate these problems, we propose a novel approach, Discriminative Attention-augmented Feature Learning Convolution Neural Network (DAF-CNN), which learns discriminative expression-related representations for FER. Firstly, we develop a 3D attention mechanism for feature refinement which selectively focuses on attentive channel entries and salient spatial regions of a convolution neural network feature map. Moreover, a deep metric loss termed Triplet-Center (TC) loss is incorporated to further enhance the discriminative power of the deeply-learned features with an expression-similarity constraint. It simultaneously minimizes intra-class distance and maximizes inter-class distance to learn both compact and separate features. Extensive experiments have been conducted on two representative facial expression datasets (FER-2013 and SFEW 2.0) to demonstrate that DAF-CNN effectively captures discriminative feature representations and achieves competitive or even superior FER performance compared to state-of-the-art FER methods.

Destruction and Reconstruction Learning for Facial Expression Recognition

The most discriminative expression features are mostly concentrated in local key facial regions. Thus, we propose a simple and efficient framework that can learn more discriminative expression features from scrambled facial images. Specifically, we first divide the input image into local subregions of the same size and shuffle them randomly at a certain range to obtain the damaged image to increase the difficulty of recognition. Then, the original image and the damaged image are fed to the network. A channel attention module is exploited for highlighting the effective features and suppressing irrelevant features. Simultaneously, during the reconstruction phase, a region alignment model is appended to establish the semantic correlation between each subregion, aiming at restoring the original spatial layout of local subregions in the original image. Extensive experiments on the RAF-DB and the FERPlus datasets demonstrate that our proposed method significantly outperformed state-of-the-art methods without any external facial expression pretraining.

Joint Expression Synthesis and Representation Learning for Facial Expression Recognition

Facial expression recognition (FER) is a challenging task due to the large appearance variations and the lack of sufficient training data. Conventional deep approaches either learn a good representation through deep models or synthesize images automatically to enlarge the training set. In this paper, we perform both tasks jointly and propose an end-to-end deep model for simultaneous facial expression recognition and facial image synthesis. The proposed model is based on Generative Adversarial Network (GAN) and enjoys several merits. First, the facial image synthesis and facial expression recognition tasks can boost their performance for each other via the unified model. Second, paired images are not required in our facial image synthesis network, which makes the proposed model much more general and flexible. Meanwhile, the generated facial images largely expand the training set and ease the overfitting problem in our FER task. Third, different expressions are encoded in a disentangled manner in a latent space, which enables us to synthesize facial images with arbitrary expressions by exchanging certain parts of their latent identity features. Quantitative and qualitative evaluations on both controlled and in-the-wild FER benchmarks (Multi-PIE, MMI, and RAF-DB) demonstrate the effectiveness of our proposed method on both facial image synthesis and facial expression recognition task.

Subspace learning for facial expression recognition: an overview and a new perspective

For image recognition, an extensive number of subspace-learning methods have been proposed to overcome the high-dimensionality problem of the features being used. In this paper, we first give an overview of the most popular and state-of-the-art subspace-learning methods, and then, a novel manifold-learning method, named soft locality preserving map (SLPM), is presented. SLPM aims to control the level of spread of the different classes, which is closely connected to the generalizability of the learned subspace. We also do an overview of the extension of manifold learning methods to deep learning by formulating the loss functions for training, and further reformulate SLPM into a soft locality preserving (SLP) loss. These loss functions are applied as an additional regularization to the learning of deep neural networks. We evaluate these subspace-learning methods, as well as their deep-learning extensions, on facial expression recognition. Experiments on four commonly used databases show that SLPM effectively reduces the dimensionality of the feature vectors and enhances the discriminative power of the extracted features. Moreover, experimental results also demonstrate that the learned deep features regularized by SLP acquire a better discriminability and generalizability for facial expression recognition.

Dynamic Objectives Learning for Facial Expression Recognition

Facial expression recognition has been widely used to solve the problems such as lie detection and human-machine interaction. However, due to the difficulties to control the application environments, current methods have the lower recognition accuracy in practice. This paper proposes a new method for facial expression recognition by considering several aspects. First, human beings are easy to recognize some expressions, while difficult to recognize others. Inspired by this intuition, a new loss function is proposed to enlarge the distances between samples from easily confused categories. Second, human learning is divided into many stages, and the learning objective of each stage is different. Thus, dynamic objectives learning is proposed, where each objective at different stage is defined by the corresponding loss function. In order to better realize the above ideas, a new deep neural network for facial expression recognition is proposed, which integrates the covariance pooling layer and residual network units into the deep convolution neural network so as to better perform dynamic objectives learning. The experimental results on the standard databases verify the effectiveness and the superior performance of our methods.