Expression Recognition Method Research Articles

Facial expression recognition sensing the complexity of testing samples

Facial expression recognition has always been a challenging issue due to the inconsistencies in the complexity of samples and variability of between expression categories. Many facial expression recognition methods train a classification model and then use this model to identify all test samples, without considering the complexity of each test sample. They are inconsistent with human cognition laws such as the principle of simplicity, so that they are easily under-learned and then are difficult to identify test samples correctly. Hence, this paper proposed a new facial expression recognition method sensing the complexity of test samples, which can nicely solve the problem of the inconsistent distribution of samples complexity. It firstly divided the training data into the hard subset and the easy subset for classification according to the complexity of samples for expression recognition. Subsequently, these two subsets are applied to train two classifiers. Instead of using the same classifier to predict all test samples, our method assigned each test sample to the corresponding classifier based on the complexity of the test sample. The experimental results demonstrated the effectiveness of the proposed method and obtained a significant improvements of the recognition performance on benchmark datasets.

A snapshot research and implementation of multimodal information fusion for data-driven emotion recognition

With the rapid development of artificial intelligence and mobile Internet, the new requirements for human-computer interaction have been put forward. The personalized emotional interaction service is a new trend in the human-computer interaction field. As a basis of emotional interaction, emotion recognition has also introduced many new advances with the development of artificial intelligence. The current research on emotion recognition mostly focuses on single-modal recognition such as expression recognition, speech recognition, limb recognition, and physiological signal recognition. However, the lack of the single-modal emotional information and vulnerability to various external factors lead to lower accuracy of emotion recognition. Therefore, multimodal information fusion for data-driven emotion recognition has been attracting the attention of researchers in the affective computing filed. This paper reviews the development background and hot spots of the data-driven multimodal emotion information fusion. Considering the real-time mental health monitoring system, the current development of multimodal emotion data sets, the multimodal features extraction, including the EEG, speech, expression, text features, and multimodal fusion strategies and recognition methods are discussed and summarized in detail. The main objective of this work is to present a clear explanation of the scientific problems and future research directions in the multimodal information fusion for data-driven emotion recognition field.

Facial expression recognition method based on deep convolutional neural network combined with improved LBP features

Aiming at the disadvantages of the traditional machine-based facial expression recognition method that eliminates the feature of manual selection, a feature extraction method based on deep convolutional neural network to learn expression features is proposed. Since the deep convolutional neural network can directly use the original image as the input image, the image abstract feature interpretation is obtained at the fully connected layer of the image, which avoids the inherent error of image preprocessing and artificial selection features. Then, we reconstruct the traditional local binary pattern (LBP) feature operator for facial expression image and fuse the abstract facial expression features learned by the deep convolution neural network with the modified LBP facial expression texture features in the full connection layer. A new facial expression feature can be obtained, and the classification accuracy can be improved. In general, for the recognition of facial expression images, the proposed method based on the fusion LBP expression features and convolutional neural network expression features is used to obtain the best performance of 91.28% in the comparative experiment. An efficient extension of the expression feature texture expression channel is carried out. On the other hand, convolutional neural networks have incomparable advantages over other methods in abstract information representation of two-dimensional images.

MDTP: a novel multi-directional triangles pattern for face expression recognition

Face expression recognition is a key-subject of machine learning, and the primary issue on it is lack of accuracy. This paper proposes a novel face expression recognition method using Decision Based Rule-Oriented Median Filter (DBROMF) and Multi-Directional Triangles Pattern (FER-MDTP). The DBROMF is a novel noise reduction method which removes the impulse noise from the facial images. This FER method enriched with a new image descriptor (MDTP) which is structured via multi-directional triangle pattern to provide a superior image description. An unparalleled novel algorithm to locate the human face organ viz. lip and eyeball stuffed in this research by getting assistance with fuzzy edge strength and abbreviated as an MDTP-FES method. These landmarks of features extracted, and the histogram oriented features tailored with the classification division. The support vector neural network classifier (SVNN) is integrated to conduct the classification job. The JAFFE, CK, TFEID, and ADFES databases are linked to perform the simulation which telescoped with six face expressions. The proposed method lifts the accuracy to a significant range than the existing state-of-the-art methods.

A novel maximum and minimum response-based Gabor (MMRG) feature extraction method for facial expression recognition

In facial expression recognition applications, the images are corrupted with random noise, and this affects the classification accuracy. This article proposes a maximum and Minimum Response-based Gabor (MMRG) that can encode the facial texture more discriminatively and eliminate random noise. Two code images are produced from the available Gabor images. Then, after dividing the code images into grids, feature vectors are formed using histograms. A technique based on the bat algorithm is proposed for the optimization of the Gabor filter banks as Bat Algorithm-based Gabor Optimization (BAGO). The MMRG increases the efficiency of Gabor filter-based features by precisely distinguishing the texture frequencies. It also helps in reducing the dimensions of feature vector which is a major problem in Gabor filter-based feature extraction. Radial Basis Function-Extreme Learning Machine (RBF-ELM) classifier is used for a faster and accurate multi-classification. The proposed approach has been evaluated with six datasets namely, Japanese Female Facial Expression (JAFFE), Cohn Kanade (CK+), Multi-media Understanding Group (MUG), Static Facial Expressions in the Wild (SFEW), Oulu-Chinese Academy of Science, Institute of Automation (Oulu-CASIA) and Man–Machine Interaction (MMI) datasets to meet a classification accuracy of 97.2, 97.4, 95.4, 35.4, 87.4 and 82.3% for seven class emotion detection, which is high when compared to other state-of –the-art methods.

A Survey on Image Acquisition Protocols for Non-posed Facial Expression Recognition Systems

Several research methodologies and human face image databases have been developed based on deliberately produced facial expressions of prototypical emotions. However, real-time and spontaneous facial expression recognition cannot be adequately handled by those existing methods and datasets. To address this problem, research efforts have been made to create spontaneous facial expression image datasets as well as to develop algorithms that can process naturally induced affective behavior. This paper introduces these advances and focuses on a small and specific area of spontaneous facial expression recognition. In this paper, we are concentrating on non-posed image acquisition protocols, which strongly influence the subjects for evoking expressions as natural as possible. We categorize the acquisition protocols into four different parts: image acquisition while playing video games, watching emotional videos, during interviews and from other sources. The taxonomy of facial expression acquisition protocols tells about the typical conditions responsible for producing specific facial expressions in that condition. We also address some important design issues related to spontaneous facial expression recognition systems and list the facial expression databases, which are strictly not acted and non-posed. We also put light on the applications of spontaneously evoked facial expression acquisition and recognition because they have potential medical significance. Moreover, we provide a comprehensive analysis and summary of spontaneous facial expression recognition methods by revealing their pros and cons for future researchers.

Appearance Based Feature Extraction and Selection Methods for Facial Expression Recognition

Faces contain a plethora of information such as identity of a person, facial expressions and other properties like race etc. Human face conveys the wealth of their opinions, emotions in the form of facial expressions. The problem of automatic facial expression is both interesting and quite challenging with a strong impact on many application areas such as animation and human-computer interaction (HCI). The field has shown a tremendous growth over past years with its benchmarking efforts and progress. In this paper, the study of facial expression recognition based feature extraction and selection techniques is presented. The paper offers an insight into the work of researchers who have considered appearance based feature extraction techniques for facial expression recognition. In addition, a discussion is made by specifying some of the open question related to facial expression recognition.

Robust and energy-efficient expression recognition based on improved deep ResNets.

To improve the robustness and to reduce the energy consumption of facial expression recognition, this study proposed a facial expression recognition method based on improved deep residual networks (ResNets). Residual learning has solved the degradation problem of deep Convolutional Neural Networks (CNNs); therefore, in theory, a ResNet can consist of infinite number of neural layers. On the one hand, ResNets benefit from better performance on artificial intelligence (AI) tasks, thanks to its deeper network structure; meanwhile, on the other hand, it faces a severe problem of energy consumption, especially on mobile devices. Hence, this study employs a novel activation function, the Noisy Softplus (NSP), to replace rectified linear units (ReLU) to get improved ResNets. NSP is a biologically plausible activation function, which was first proposed in training Spiking Neural Networks (SNNs); thus, NSP-trained models can be directly implemented on ultra-low-power neuromorphic hardware. We built an 18-layered ResNet using NSP to perform facial expression recognition across datasets Cohn-Kanade (CK+), Karolinska Directed Emotional Faces (KDEF) and GENKI-4K. The results achieved better anti-noise ability than ResNet using the activation function ReLU and showed low energy consumption running on neuromorphic hardware. This study not only contributes a solution for robust facial expression recognition, but also consolidates the low energy cost of their implementation on neuromorphic devices, which could pave the way for high-performance, noise-robust and energy-efficient vision applications on mobile hardware.

Facial expression recognition algorithm based on parameter adaptive initialization of CNN and LSTM

In view of the high dimensionality, nonrigidity, multiscale variation and the influence of illumination and angle on facial expressions, it is quite difficult to obtain facial expression images or videos using computers and analyze facial morphology and changes to accurately obtain the emotional changes of the subjects. Existing facial expression recognition algorithms have the following problems in the application process: the existing shallow feature extraction model has lost a lot of effective feature information and low recognition accuracy. The facial expression recognition method based on deep learning has problems such as overfitting, gradient explosion and parameter initialization. Therefore, this paper develops a facial expression recognition algorithm based on the deep learning method. An adaptive model parameter initialization based on the multilayer maxout network linear activation function is proposed to initialize the convolutional neural network (CNN) and the long–short-term memory network (LSTM) method. It can effectively overcome the gradient disappearance and gradient explosion problems in the deep learning model training process. At the same time, the convolutional neural network with an LSTM memory unit is used to extract the related information from the image sequence, and the facial expression judgment is based on a single-frame image and historical-related information. However, the top-level structure of the CNN model is a fully connected feedforward neural network, which undertakes the task of expression classification. Therefore, the SVM classification method replaces the top-level classifier to further improve the expression classification accuracy. Experiments show that the facial expression recognition method proposed in this paper not only accurately identifies various expressions but also has good adaptive ability. This is because the method achieves the adaptive initialization of the parameters of the deep learning model construction process and also analyzes the relevance of the expression database expression, thereby improving the accuracy of expression recognition.

Sparse modified marginal fisher analysis for facial expression recognition

Marginal Fisher analysis (MFA) is an efficient method for dimension reduction, which can extract useful discriminant features for image recognition. Since sparse learning can achieve better generalization ability and lessen the amount of computations in recognition tasks, this paper introduces sparsity into MFA and proposes a novel sparse modified MFA (SMMFA) method for facial expression recognition. The goal of SMMFA is to extract discriminative features by using the resulted sparse projection matrix. First, a modified MFA is proposed to find the original projection matrix. Similar to MFA, the modified MFA also defines the intra-class graph and the inter-class graph to describe geometry structure in the same class and local discriminant structure between different classes, respectively. In addition, the modified MFA removes the null space of the total scatter matrix. The sparse solution of SMMFA can be gained by solving the l1 –minimization problem on the original projection matrix using the linearized Bregman iteration. Experimental results show that the proposed SMMFA can effectively extract intrinsic features and has better discriminant power than the state-of-the-art methods.

Local Appearance based Novel Facial Feature Extraction Method for Human Expression Recognition

Local Appearance based Novel Facial Feature Extraction Method for Human Expression Recognition

Facial expression recognition with local prominent directional pattern

Local edge-based descriptors have gained much attention as feature extraction methods for facial expression recognition. However, such descriptors are found to suffer from unstable shape representations for different local structures for their sensitivity to local distortions such as noise and positional variations. We propose a novel edge-based descriptor, named Local Prominent Directional Pattern (LPDP), which considers statistical information of a pixel neighborhood to encode more meaningful and reliable information than the existing descriptors for feature extraction. More specifically, LPDP examines a local neighborhood of a pixel to retrieve significant edges corresponding to the local shape and thereby ensures encoding edge information in spite of some positional variations and avoiding noisy edges. Thus LPDP can represent important textured regions much effectively to be used in facial expression recognition. Extensive experiments on facial expression recognition on well-known datasets also demonstrate the better capability of LPDP than other existing descriptors in terms of robustness in extracting various local structures originated by facial expression changes.

FERCaps: A Capsule-Based Method for Face Expression Recognition from Frontal Face Images

A novel method which is named FERCaps (capsule-based method for facial expression recognition) is presented in this paper. We process these images containing the faces, and segment these gray images only containing the front faces for training the model. The model framework consists three layers of convolutional neural networks (CNN), two layers of capsules, and a decoder with a fully connected layer and four deconvolution. The ReLU activation function is used to speed up the experimental model training speed in the convolution layers and constructed a decoder suitable for facial expressions using deconvolution. The original capsule is the bottom layer of the multidimensional entity. Through an iterative routing process, each raw capsule selects a corresponding high-level capsule in the upper level. Instantiated parameters can characterize facial expressions in high-level capsules. We verified the effectiveness of the model through experiments on the public benchmarking datasets JAFFE and extended Cohn-Kanade (CK+), and achieved an accuracy of 98.18% in ck+ and 88.33% in JAFFE

A Face Emotion Recognition Method Using Convolutional Neural Network and Image Edge Computing

To avoid the complex process of explicit feature extraction in traditional facial expression recognition, a face expression recognition method based on a convolutional neural network (CNN) and an image edge detection is proposed. Firstly, the facial expression image is normalized, and the edge of each layer of the image is extracted in the convolution process. The extracted edge information is superimposed on each feature image to preserve the edge structure information of the texture image. Then, the dimensionality reduction of the extracted implicit features is processed by the maximum pooling method. Finally, the expression of the test sample image is classified and recognized by using a Softmax classifier. To verify the robustness of this method for facial expression recognition under a complex background, a simulation experiment is designed by scientifically mixing the Fer-2013 facial expression database with the LFW data set. The experimental results show that the proposed algorithm can achieve an average recognition rate of 88.56% with fewer iterations, and the training speed on the training set is about 1.5 times faster than that on the contrast algorithm.

Reliable Crowdsourcing and Deep Locality-Preserving Learning for Unconstrained Facial Expression Recognition.

Facial expression is central to human experience, but most previous databases and studies are limited to posed facial behavior under controlled conditions. In this paper, we present a novel facial expression database, Real-world Affective Face Database (RAF-DB), which contains approximately 30 000 facial images with uncontrolled poses and illumination from thousands of individuals of diverse ages and races. During the crowdsourcing annotation, each image is independently labeled by approximately 40 annotators. An expectation-maximization algorithm is developed to reliably estimate the emotion labels, which reveals that real-world faces often express compound or even mixture emotions. A cross-database study between RAF-DB and CK+ database further indicates that the action units of real-world emotions are much more diverse than, or even deviate from, those of laboratory-controlled emotions. To address the recognition of multi-modal expressions in the wild, we propose a new deep locality-preserving convolutional neural network (DLP-CNN) method that aims to enhance the discriminative power of deep features by preserving the locality closeness while maximizing the inter-class scatter. Benchmark experiments on 7-class basic expressions and 11-class compound expressions, as well as additional experiments on CK+, MMI, and SFEW 2.0 databases, show that the proposed DLP-CNN outperforms the state-of-the-art handcrafted features and deep learning-based methods for expression recognition in the wild. To promote further study, we have made the RAF database, benchmarks, and descriptor encodings publicly available to the research community.

Facial expression recognition using iterative fusion of MO-HOG and deep features

Facial expression recognition is a challenging problem in computer vision. Due to the limited feature extraction capability of a single feature descriptor, this paper proposes a facial expression recognition method that iteratively fuses classifiers based on multi-orientation gradient calculated HOG (MO-HOG) features and deep-learned features. Diagonal orientation gradient calculated HOG (D-HOG) is a complementary part to the histogram of oriented gradient (HOG), which is proposed to obtain the diagonal gradient information and combines HOG to form a novel feature descriptor MO-HOG. Our method extracts MO-HOG features from whole facial images and expression-rich local facial images. Meanwhile, deep-learned features are not reliable enough on small databases but contain high-level semantic information, so the deep network is designed to extract effective deep-learned features. In addition, a classifier fusion method based on an optimization algorithm is proposed, and the best-fused classifier is obtained through iteration. The experiments are evaluated on the public databases (CK+ and JAFFE). The proposed method shows the effectiveness of facial expression recognition and outperforms the state-of-the-art methods. The recognition accuracy is 97.70% on the CK+ database and 97.64% on the JAFFE database.

Expression-targeted feature learning for effective facial expression recognition

In this paper, we propose a novel expression-targeted feature learning (ETFL) method for effective facial expression recognition, which takes advantage of multi-task learning for discriminative feature learning. Specifically, the common features are firstly extracted from the lower layers of CNN. Then, based on the common features, the expression-specific features (ESF) are respectively learned for each facial expression via multi-task learning. In order to enhance the discriminability of ESF, we develop a joint loss (the combination of the center loss and a novel inter-class loss) to explicitly reduce intra-class variations while enlarging inter-class differences. Furthermore, we introduce the sample-sensitive weights and the soft-expression weights to balance the joint loss for better performance. Finally, all ESFs are combined for final classification. ETFL effectively exploits the relationship among all facial expressions, which leads to superiority feature discriminability. Experiments on public facial expression databases demonstrate the effectiveness of ETFL compared with several state-of-the-art methods.

Thermal Augmented Expression Recognition.

Visible facial images provide geometric and appearance patterns of facial expressions and are sensitive to illumination changes. Thermal facial images record facial temperature distribution and are robust to light conditions. Therefore, expression recognition is enhanced by visible and thermal image fusion. In most cases, only visible images are available due to the widespread popularity of visible cameras and the high cost of thermal cameras. Thus, we propose a novel visible expression recognition method by using thermal infrared (IR) data as privileged information, which is only available during training. Specifically, we first learn a deep model for visible images and thermal images. Then we use the learned feature representations to train support vector machine (SVM) classifiers for expression classification. We jointly refine the deep models as well as the SVM classifiers for both thermal images and visible images by imposing the constraint that the outputs of the SVM classifiers from two views are similar. Thermal IR images during training are then exploited to construct better facial representations and expression classifiers from visible images. We extend the proposed thermal augmented expression recognition method for partially unpaired data, acknowledging that visible images and thermal images maybe not be recorded synchronously. Experimental resulton the MAHNOB laughter database demonstrate that the proposed thermal augmented expression recognition method can effectively exploit thermal IR images' supplementary role for visible facial expression recognition during training to obtain better facial representations and a better visible expression classifier. The proposed thermal augmented expression recognition method achieves state-of-the-art expression recognition performance for both paired and unpaired facial images.

Robust facial expression recognition using RGB-D images and multichannel features

Traditional and classical methods of facial expression recognition are mainly based on intensity image and are prone to be disturbed by illumination, poses, and disguise. This research aims to develop a robust facial expression recognition method using RGB-D images and multichannel features. Based on image entropy and visual saliency, facial texture features are firstly extracted from RGB images and depth images to construct the Histogram of Oriented Gradient (HOG) descriptors. And then, we extract geometric features of RGB images using Active Appearance Model (AAM). Combining the HOG texture features with the AAM geometric feature, we build a robust multichannel feature vector for facial expression recognition. On this basis, an improved Support Vector Machine (SVM) algorithm, namely GS-SVM, is used to classify facial expression recognition. The proposed GS-SVM algorithm applies Grid Search method to optimize the best parameters for SVM classifier and estimate the accuracy of each parameter combination in specified range. Finally, the proposed methods are tested and evaluated on the merged RGB-D database. Experimental results show that the proposed algorithm not only achieves a higher average recognition rate but also is robust to uncontrolled environments.

A visual attention based ROI detection method for facial expression recognition

Abstract In this paper, an eleven-layered Convolutional Neural Network with Visual Attention is proposed for facial expression recognition. The network is composed of three components. First, local convolutional features of faces are extracted by a stack of ten convolutional layers. Second, the regions of interest are automatically determined according to these local features by the embedded attention model. Third, the local features in these regions are aggregated and used to infer the emotional label. These three components are integrated into a single network which can be trained in an end-to-end scheme. Extensive experiments on four kinds of data (namely aligned frontal faces, faces in different poses, aligned unconstrained faces, and grouped unconstrained faces) prove that the proposed method can improve the accuracy and obtain good visualization. The visualization shows that the learned regions of interest are partly consistent with the locations of emotion specific Action Units. This founding confirms the interpretation of Facial Action Coding System and Emotional Facial Action Coding System from a machine learning perspective.