Multi-view Facial Expression Recognition Research Articles

Contrastive Learning of View-invariant Representations for Facial Expressions Recognition

Although there has been much progress in the area of facial expression recognition (FER), most existing methods suffer when presented with images that have been captured from viewing angles that are non-frontal and substantially different from those used in the training process. In this article, we propose ViewFX, a novel view-invariant FER framework based on contrastive learning, capable of accurately classifying facial expressions regardless of the input viewing angles during inference. ViewFX learns view-invariant features of expression using a proposed self-supervised contrastive loss, which brings together different views of the same subject with a particular expression in the embedding space. We also introduce a supervised contrastive loss to push the learned view-invariant features of each expression away from other expressions. Since facial expressions are often distinguished with very subtle differences in the learned feature space, we incorporate the Barlow twins loss to reduce the redundancy and correlations of the representations in the learned representations. The proposed method is a substantial extension of our previously proposed CL-MEx, which only had a self-supervised loss. We test the proposed framework on two public multi-view facial expression recognition datasets, KDEF and DDCF. The experiments demonstrate that our approach outperforms previous works in the area and sets a new state-of-the-art for both datasets while showing considerably less sensitivity to challenging angles and the number of output labels used for training. We also perform detailed sensitivity and ablation experiments to evaluate the impact of different components of our model as well as its sensitivity to different parameters.

MFERMeta++ : Robust Multiview Facial Expression Recognition Based on Metahuman and Metalearning

Facial pose variation presents a significant challenge to facial expression recognition (FER) in real‐world applications. Significant bottlenecks exist in the field of multiview facial expression recognition (MFER) including a lack of high‐quality MFER datasets, and the limited model robustness in real‐world MFER scenarios. Therefore, this article first introduces a metahuman‐based MFER dataset (MMED), which effectively addresses the issues of insufficient quantity and quality in existing datasets. Second, a conditional cascade VGG (ccVGG) model is proposed, which can adaptively adjust expression feature extraction based on the input image's pose information. Finally, a hybrid training and few‐shot learning strategy are proposed that integrates our MMED dataset with a real‐world dataset and quickly deploys it in real‐world application scenarios using the proposed Meta‐Dist few‐shot learning method. Experiments on the Karolinska Directed Emotional Face (KDEF) dataset demonstrate that the proposed model exhibits improved robustness in multiview application scenarios and achieves a recognition accuracy improvement of 28.68% relative to the baseline. It demonstrates that the proposed MMED dataset can effectively improve the training efficiency of MFER models and facilitate easy deployment in real‐world applications. This work provides a reliable dataset for the MFER studies and paves the way for robust FER in any view for real‐world deployment.

Joint spatial and scale attention network for multi-view facial expression recognition

Multi-view facial expression recognition (FER) is a challenging task because the appearance of an expression varies greatly due to poses. To alleviate the influences of poses, recently developed methods perform pose normalization, learn pose-invariant features, or learn pose-specific FER classifiers. However, these methods usually rely on a prerequisite pose estimator or expressive region detector that is independent of the subsequent expression analysis. Different from existing methods, we propose a joint spatial and scale attention network (SSA-Net) to localize proper regions for simultaneous head pose estimation (HPE) and FER. Specifically, SSA-Net discovers the regions most relevant to the facial expression at hierarchical scales by a spatial attention mechanism, and the most informative scales are selected in a scale attention learning manner to learn the joint pose-invariant and expression-discriminative representations. Then, we employ a dynamically constrained multi-task learning mechanism with a delicately designed constrain regulation to properly and adaptively train the network to optimize the representations, thus achieving accurate multi-view FER. The effectiveness of the proposed SSA-Net is validated on three multi-view datasets (BU-3DFE, Multi-PIE, and KDEF) and three in-the-wild FER datasets (AffectNet, SFEW, and FER2013). Extensive experiments demonstrate that the proposed framework outperforms existing state-of-the-art methods under both within-dataset and cross-dataset settings, with relative accuracy gains of 2.36%, 1.33%, 3.11%, 2.84%, 15.7%, and 7.57%, respectively.

Multiview Facial Expression Recognition, A Survey

Multiview Facial Expression Recognition (MFER) is a well-known interdisciplinary problem among computer science and related disciplines with promising and valuable applications. Recognizing the facial expression in pose variations, which is very common in real-world conditions, makes it very challenging. This paper aims to provide a comprehensive survey of the MFER progress, includingboth categories of traditional and deep approaches. In general, we sort each of these categories into three overall groups to meet the pose variations: Pose-Robust Features, Pose Normalization, and Pose-Specific Classification. While reviewing the traditional methods, a thorough study is proposed on the existing novel deep techniques. We also introduce the state-of-the-art and discuss the challenges, limitations, opportunities, and future trends that need to be addressed in this field. Moreover, we provide an extensive review of publicly available datasets for MFER, including the labs’ collections and the sets gathered from in the wild. Besides, we introduce the most popular protocols on each dataset to standardize comparisons in the future.

Orthogonal channel attention-based multi-task learning for multi-view facial expression recognition

Multi-view facial expression recognition (FER) is a challenging computer vision task due to the large intra-class difference caused by viewpoint variations. This paper presents a novel orthogonal channel attention-based multi-task learning (OCA-MTL) approach for FER. The proposed OCA-MTL approach adopts a Siamese convolutional neural network (CNN) to force the multi-view expression recognition model to learn the same features as the frontal expression recognition model. To further enhance the recognition accuracy of non-frontal expression, the multi-view expression model adopts a multi-task learning framework that regards head pose estimation (HPE) as an auxiliary task. A separated channel attention (SCA) module is embedded in the multi-task learning framework to generate individual attention for FER and HPE. Furthermore, orthogonal channel attention loss is presented to force the model to employ different feature channels to represent the facial expression and head pose, thereby decoupling them. The proposed approach is performed on two public facial expression datasets to evaluate its effectiveness and achieves an average recognition accuracy rate of 88.41% under 13 viewpoints on Multi-PIE and 89.04% under 5 viewpoints on KDEF, outperforming state-of-the-art methods.

GAN Based Three-Stage-Training Algorithm for Multi-view Facial Expression Recognition

GAN Based Three-Stage-Training Algorithm for Multi-view Facial Expression Recognition

A Joint Mapping and Synthesis Approach for Multiview Facial Expression Recognition

This paper presents a novel approach to address pose-invariant face frontalization aiming Multiview Facial Expression Recognition (MFER). Particularly, the proposed approach is a hybrid method, including both synthesizing and mapping techniques. The key idea is to use mapped reconstructive coefficients of each arbitrary viewpoints and the frontal bases where the mapping functions are provided by learning between frontal and non-frontal faces’ coefficients. We also exploit sparse coding for synthesizing the frontalized faces, even with large poses. For evaluation, two qualitative and quantitative assessments are used along with an application of multiview facial expression recognition as a case study. The results show that our approach is efficient in terms of frontalizing non-frontal faces. Moreover, its validation on two popular datasets, BU3DFE and Multi-PIE, with various assessments contexts reveals its efficiency and stability on head pose variation, especially on large poses.

Improving multi-view facial expression recognition through two novel texture-based feature representations

Although several automatic computer systems have been proposed to address facial expression recognition problems, the majority of them still fail to cope with some requirements of many practical application scenarios. In this paper, one of the most influential and common issues raised in practical application scenarios when applying automatic facial expression recognition system, head pose variation, is comprehensively explored and investigated. In order to do this, two novel texture feature representations are proposed for implementing multi-view facial expression recognition systems in practical environments. These representations combine the block-based techniques with Local Ternary Pattern-based features, providing a more informative and efficient feature representation of the facial images. In addition, an in-house multi-view facial expression database has been designed and collected to allow us to conduct a detailed research study of the effect of out-of-plane pose angles on the performance of a multi-view facial expression recognition system. Along with the proposed in-house dataset, the proposed system is tested on two well-known facial expression databases, CK+ and BU-3DFE datasets. The obtained results shows that the proposed system outperforms current state-of-the-art 2D facial expression systems in the presence of pose variations.

Multi-View Facial Expression Recognition with Multi-View Facial Expression Light Weight Network

Facial expression recognition for frontal faces has become a well-established research area in the last two decades. However, non-frontal facial expression recognition hasn’t been paid much attention until recently. In this paper, we propose an MVFE-LightNet (Multi-View Facial Expression Light Weight Network) for multi-view facial expression recognition. To this end, we first applied MTCNN for facial detection and alignment and then did preprocessing like normalization and data augmentation. Finally, we put the images into MVFE-LightNet to extract sub-space features of facial expressions with various poses. A depthwise separable residual convolution module architecture was designed to reduce the parameters of the model and lessen the chance of overfitting. Experiments were implemented on Radboud Faces Database and BU-3DFE dataset. We demonstrated that our method could effectively improve the recognition accuracy, and achieved the accuracy of 95.6% and 88.7% respectively for the Radboud and BU-3DFE.

Multi-level uncorrelated discriminative shared Gaussian process for multi-view facial expression recognition

In multi-view facial expression recognition, discriminative shared Gaussian process latent variable model (DS-GPLVM) gives better performance than that of linear and nonlinear multi-view learning-based methods. However, Laplacian-based prior used in DS-GPLVM only captures topological structure of data space without considering the inter-class separability of the data, and hence the obtained latent space is suboptimal. So, we propose a multi-level uncorrelated DS-GPLVM (ML-UDSGPLVM) model which searches a common uncorrelated discriminative latent space learned from multiple observable spaces. A novel prior is proposed, which not only depends on the topological structure of the intra-class data, but also on the local-between-class-scatter-matrix of the data onto the latent manifold. The proposed approach employs an hierarchical framework, in which, expressions are first divided into three sub-categories. Subsequently, each of the sub-categories are further classified to identify the constituent basic expressions. Experimental results show that the proposed method outperforms state-of-the-art methods in many instances.

Hierarchical uncorrelated multiview discriminant locality preserving projection for multiview facial expression recognition

Existing multi-view facial expression recognition algorithms are not fully capable of finding discriminative directions if the data exhibits multi-modal characteristics. This research moves toward addressing this issue in the context of multi-view facial expression recognition. For multi-modal data, local preserving projection (LPP) or local Fisher discriminant analysis (LFDA)-based approach is quite appropriate to find a discriminative space. Also, the classification performance can be enhanced by imposing uncorrelated constraint onto the discriminative space. So for multi-view (multi-modal) data, we proposed an uncorrelated multi-view discriminant locality preserving projection (UMvDLPP)-based approach to find an uncorrelated common discriminative space. Additionally, the proposed UMvDLPP is implemented in a hierarchical fashion (H-UMvDLPP) to obtain an optimal performance. Extensive experiments on BU3DFE dataset show that UMvDLPP performs slightly better than the existing methods. However, an improvement of approximately 3% as compared to the existing state-of-the-art multi-view learning-based approaches is achieved by our H-UMvDLPP. This improvement is due to the fact that the proposed method enhances the discrimination between the classes more effectively, and classifies expressions category-wise followed by classification of the basic expressions embedded in each of the subcategories (hierarchical approach).

Mapping Forests: A Comprehensive Approach for Nonlinear Mapping Problems

A new and robust mapping approach is proposed entitled mapping forests (MFs) for computer vision applications based on regression transformations. Mapping forests relies on learning nonlinear mappings deduced from pairs of source and target training data, and improves the performance of mappings by enabling nonlinear transformations using forests. In contrast to previous approaches, it provides automatically selected mappings, which are naturally nonlinear. MF can provide accurate nonlinear transformations to compensate the gap of linear mappings or can generalize the nonlinear mappings with constraint kernels. In our experiments, we demonstrate that the proposed MF approach is not only on a par or better than linear mapping approaches and the state-of-the-art, but also is very time efficient, which makes it an attractive choice for real-time applications. We evaluated the efficiency and performance of the MF approach using the BU3DFE and Multi-PIE datasets for multi-view facial expression recognition application, and Set5, Set14 and SuperTex136 datasets for single image super-resolution application.

A Multi-view Facial Expression Recognition Method Based on Discriminative Shared Gaussian Process Latent Variable Model

The traditional multi-view facial expression recognition method is adopted view-specific classifier to recognition view-specific sample. This approach ignores the fact that different views of a facial expression are just different manifestations of the same facial expression. To address this, a multi-view facial expression recognition method based on Discriminative Shared Gaussian Process Latent Variable Model is proposed. Firstly this method extraction Incremental Update Parallel Cascade of Linear Regression feature, then its uses PCA to select the feature, finally adopts Discriminative Shared Gaussian Process Latent Variable Model to recognition Multi-view facial expression. The experiment carried on CMU-PIE database and LFPW database show the effectiveness of our method.

Locality-constrained linear coding based bi-layer model for multi-view facial expression recognition

Multi-view facial expression recognition is a challenging and active research area in computer vision. In this paper, we propose a simple yet effective method, called the locality-constrained linear coding based bi-layer (LLCBL) model, to learn discriminative representation for multi-view facial expression recognition. To address the issue of large pose variations, locality-constrained linear coding is adopted to construct an overall bag-of-features model, which is then used to extract overall features as well as estimate poses in the first layer. In the second layer, we establish one specific view-dependent model for each view, respectively. After the pose information of the facial image is known, we use the corresponding view-dependent model in the second layer to further extract features. By combining all the features in these two layers, we obtain a unified representation of the image. To evaluate the proposed approach, we conduct extensive experiments on both BU-3DFE and Multi-PIE databases. Experimental results show that our approach outperforms the state-of-the-art methods.

A Deep Neural Network-Driven Feature Learning Method for Multi-view Facial Expression Recognition

In this paper, a novel deep neural network (DNN)-driven feature learning method is proposed and applied to multi-view facial expression recognition (FER). In this method, scale invariant feature transform (SIFT) features corresponding to a set of landmark points are first extracted from each facial image. Then, a feature matrix consisting of the extracted SIFT feature vectors is used as input data and sent to a well-designed DNN model for learning optimal discriminative features for expression classification. The proposed DNN model employs several layers to characterize the corresponding relationship between the SIFT feature vectors and their corresponding high-level semantic information. By training the DNN model, we are able to learn a set of optimal features that are well suitable for classifying the facial expressions across different facial views. To evaluate the effectiveness of the proposed method, two nonfrontal facial expression databases, namely BU-3DFE and Multi-PIE, are respectively used to testify our method and the experimental results show that our algorithm outperforms the state-of-the-art methods.

Pose-specific non-linear mappings in feature space towards multiview facial expression recognition

We introduce a novel approach to recognizing facial expressions over a large range of head poses. Like previous approaches, we map the features extracted from the input image to the corresponding features of the face with the same facial expression but seen in a frontal view. This allows us to collect all training data into a common referential and therefore benefit from more data to learn to recognize the expressions. However, by contrast with such previous work, our mapping depends on the pose of the input image: We first estimate the pose of the head in the input image, and then apply the mapping specifically learned for this pose. The features after mapping are therefore much more reliable for recognition purposes. In addition, we introduce a non-linear form for the mapping of the features, and we show that it is robust to occasional mistakes made by the pose estimation stage. We evaluate our approach with extensive experiments on two protocols of the BU3DFE and Multi-PIE datasets, and show that it outperforms the state-of-the-art on both datasets.

Supervised super-vector encoding for facial expression recognition

Expression recognition from faces with varying pose and illumination conditions is a challenging research area with growing interest. In this paper, we develop a novel supervised super-vector encoding framework to learn discriminative image feature representations. The framework is then validated on the Multi-PIE and BU3D-FE databases for multi-view facial expression recognition. Extensive experiments show that our supervised framework gives significant improvement over the unsupervised counterpart and outperforms the state-of-the-arts.

Multi-View Facial Expression Recognition Based on Group Sparse Reduced-Rank Regression

In this paper, a novel multi-view facial expression recognition method is presented. Different from most of the facial expression methods that use one view of facial feature vectors in the expression recognition, we synthesize multi-view facial feature vectors and combine them to this goal. In the facial feature extraction, we use the grids with multi-scale sizes to partition each facial image into a set of sub regions and carry out the feature extraction in each sub region. To deal with the prediction of expressions, we propose a novel group sparse reduced-rank regression (GSRRR) model to describe the relationship between the multi-view facial feature vectors and the corresponding expression class label vectors. The group sparsity of GSRRR enables us to automatically select the optimal sub regions of a face that contribute most to the expression recognition. To solve the optimization problem of GSRRR, we propose an efficient algorithm using inexact augmented Lagrangian multiplier (ALM) approach. Finally, we conduct extensive experiments on both BU-3DFE and Multi-PIE facial expression databases to evaluate the recognition performance of the proposed method. The experimental results confirm better recognition performance of the proposed method compared with the state of the art methods.

Local binary patterns for multi-view facial expression recognition

Research into facial expression recognition has predominantly been applied to face images at frontal view only. Some attempts have been made to produce pose invariant facial expression classifiers. However, most of these attempts have only considered yaw variations of up to 45°, where all of the face is visible. Little work has been carried out to investigate the intrinsic potential of different poses for facial expression recognition. This is largely due to the databases available, which typically capture frontal view face images only. Recent databases, BU3DFE and multi-pie, allows empirical investigation of facial expression recognition for different viewing angles. A sequential 2 stage approach is taken for pose classification and view dependent facial expression classification to investigate the effects of yaw variations from frontal to profile views. Local binary patterns ( LBPs) and variations of LBPs as texture descriptors are investigated. Such features allow investigation of the influence of orientation and multi-resolution analysis for multi-view facial expression recognition. The influence of pose on different facial expressions is investigated. Others factors are investigated including resolution and construction of global and local feature vectors. An appearance based approach is adopted by dividing images into sub-blocks coarsely aligned over the face. Feature vectors contain concatenated feature histograms built from each sub-block. Multi-class support vector machines are adopted to learn pose and pose dependent facial expression classifiers.