Robust Facial Expression Recognition Research Articles

Robust facial expression recognition system based on hidden Markov models

Facial expressions are considered important communicative tools. In this paper, we present a new system able to recognize facial expressions. This system utilizes one-dimensional Hidden Markov Models (1D-HMMs) with small number of states involving fast computational time during the learning and recognition steps. First, face features are obtained using Gabor wavelets. Then, the Fisher’s Discriminant Analysis method is employed to reduce the features dimensions to eliminate redundancy and overlap in the information. The proposed system differs from previous methods using 1D-HMMs, by the fact that it employs 1D-HMMs with only three states without any prior segmentation step of interest regions in the face (hair, forehead, eyes, etc). The proposed system is evaluated using the JAFFE and KDEF facial expressions data sets.

Robust facial expression recognition algorithm based on local metric learning

In facial expression recognition tasks, different facial expressions are often confused with each other. Motivated by the fact that a learned metric can significantly improve the accuracy of classification, a facial expression recognition algorithm based on local metric learning is proposed. First, k -nearest neighbors of the given testing sample are determined from the total training data. Second, chunklets are selected from the k -nearest neighbors. Finally, the optimal transformation matrix is computed by maximizing the total variance between different chunklets and minimizing the total variance of instances in the same chunklet. The proposed algorithm can find the suitable distance metric for every testing sample and improve the performance on facial expression recognition. Furthermore, the proposed algorithm can be used for vector-based and matrix-based facial expression recognition. Experimental results demonstrate that the proposed algorithm could achieve higher recognition rates and be more robust than baseline algorithms on the JAFFE, CK, and RaFD databases.

Hierarchical committee of deep convolutional neural networks for robust facial expression recognition

This paper describes our approach towards robust facial expression recognition (FER) for the third Emotion Recognition in the Wild (EmotiW2015) challenge. We train multiple deep convolutional neural networks (deep CNNs) as committee members and combine their decisions. To improve this committee of deep CNNs, we present two strategies: (1) in order to obtain diverse decisions from deep CNNs, we vary network architecture, input normalization, and random weight initialization in training these deep models, and (2) in order to form a better committee in structural and decisional aspects, we construct a hierarchical architecture of the committee with exponentially-weighted decision fusion. In solving a seven-class problem of static FER in the wild for the EmotiW2015, we achieve a test accuracy of 61.6 %. Moreover, on other public FER databases, our hierarchical committee of deep CNNs yields superior performance, outperforming or competing with state-of-the-art results for these databases.

Human facial expression recognition using stepwise linear discriminant analysis and hidden conditional random fields.

This paper introduces an accurate and robust facial expression recognition (FER) system. For feature extraction, the proposed FER system employs stepwise linear discriminant analysis (SWLDA). SWLDA focuses on selecting the localized features from the expression frames using the partial F-test values, thereby reducing the within class variance and increasing the low between variance among different expression classes. For recognition, the hidden conditional random fields (HCRFs) model is utilized. HCRF is capable of approximating a complex distribution using a mixture of Gaussian density functions. To achieve optimum results, the system employs a hierarchical recognition strategy. Under these settings, expressions are divided into three categories based on parts of the face that contribute most toward an expression. During recognition, at the first level, SWLDA and HCRF are employed to recognize the expression category; whereas, at the second level, the label for the expression within the recognized category is determined using a separate set of SWLDA and HCRF, trained just for that category. In order to validate the system, four publicly available data sets were used, and a total of four experiments were performed. The weighted average recognition rate for the proposed FER approach was 96.37% across the four different data sets, which is a significant improvement in contrast to the existing FER methods.

Robust facial expression recognition using improved sparse classifier

Recently, sparse classifier SC has become a promising classification technique and is increasingly attracting attention in signal processing, computer vision and pattern recognition. In this paper, a new classification algorithm based on a weighted sparse representation model, called improved sparse classifier, is proposed for robust facial expression recognition. The effectiveness and robustness of the proposed method is investigated on clean and occluded facial expression images. The performance of the proposed method on robust facial expression recognition is compared with SC, the nearest neighbour NN, linear support vector machines SVM and the nearest subspace NS. Experimental results on two benchmarking facial expression databases, i.e., the JAFFE database and the Cohn-Kanade database, demonstrate that the proposed method obtains promising performance and a strong robustness to corruption and occlusion on robust facial expression recognition tasks.

Facial expression recognition using active contour-based face detection, facial movement-based feature extraction, and non-linear feature selection

Knowledge about people's emotions can serve as an important context for automatic service delivery in context-aware systems. Hence, human facial expression recognition (FER) has emerged as an important research area over the last two decades. To accurately recognize expressions, FER systems require automatic face detection followed by the extraction of robust features from important facial parts. Furthermore, the process should be less susceptible to the presence of noise, such as different lighting conditions and variations in facial characteristics of subjects. Accordingly, this work implements a robust FER system, capable of providing high recognition accuracy even in the presence of aforementioned variations. The system uses an unsupervised technique based on active contour model for automatic face detection and extraction. In this model, a combination of two energy functions: Chan---Vese energy and Bhattacharyya distance functions are employed to minimize the dissimilarities within a face and maximize the distance between the face and the background. Next, noise reduction is achieved by means of wavelet decomposition, followed by the extraction of facial movement features using optical flow. These features reflect facial muscle movements which signify static, dynamic, geometric, and appearance characteristics of facial expressions. Post-feature extraction, feature selection, is performed using Stepwise Linear Discriminant Analysis, which is more robust in contrast to previously employed feature selection methods for FER. Finally, expressions are recognized using trained HMM(s). To show the robustness of the proposed system, unlike most of the previous works, which were evaluated using a single dataset, performance of the proposed system is assessed in a large-scale experimentation using five publicly available different datasets. The weighted average recognition rate across these datasets indicates the success of employing the proposed system for FER.

Random Gabor based templates for facial expression recognition in images with facial occlusion

Robust facial expression recognition (FER) under occluded face conditions is challenging. It requires robust algorithms of feature extraction and investigations into the effects of different types of occlusion on the recognition performance to gain insight. Previous FER studies in this area have been limited. They have spanned recovery strategies for loss of texture features in local facial regions and testing limited to only a few types of occlusion and predominantly a matched train-test strategy. This paper proposes a robust approach that employs a Monte Carlo algorithm to extract a set of Gabor based part-face templates from gallery images and converts these templates into template match distance features. The resulting feature vectors are robust to occlusion because occluded parts are covered by some but not all of the random templates. The method is evaluated using facial images with occluded regions around the eyes and the mouth, randomly placed occlusion patches of different sizes, and near-realistic occlusion of eyes with clear and solid glasses. Both matched and mis-matched train and test strategies are adopted to analyze the effects of such occlusion. Overall recognition performance and the performance for each facial expression are investigated. Experimental results on the Cohn-Kanade and JAFFE databases demonstrate the high robustness and fast processing speed of our approach, and provide useful insight into the effects of occlusion on FER. Results on the parameter sensitivity experiments demonstrate a certain level of robustness of the approach to changes in the orientation and scale of Gabor filters, the size of templates, and occlusion ratios. Performance comparisons with previous approaches show that the proposed method is more robust to occlusion with lower reductions in accuracy from occlusion of eyes or mouth.

Learning realistic facial expressions from web images

A large amount of labeled training data is required to develop effective and robust facial expression analysis methods. However, obtaining such data is typically a tedious and time-consuming task. With a rapid advance of the Internet and Web technologies, it has been feasible to collect a large number of images with label information at a low cost of human efforts. In this paper, we propose a search based framework to collect realistic facial expression images from the Web so as to further advance research on robust facial expression recognition. Due to the limitation of current commercial web search engines, a large fraction of returned images is not related to a given query keyword. We present a Support Vector Machine (SVM) based active learning approach for selecting relevant images from noisy image search results. The resulting dataset is more diverse with more sample images per expression compared to other well established facial expression datasets such as CK and JAFFE. In addition, a novel facial expression feature based on the state-of-the-art Weber Local Descriptor (WLD) and histogram contextualization is proposed to handle such a challenging dataset. Comprehensive experimental results demonstrate that our web based dataset is capable of resembling more closely to the real world conditions compared to the CK and JAFFE datasets, and our proposed feature is more effective than the existing widely used features.

Robust facial expression recognition of a speaker using thermal image processing and updating of fundamental training data

We previously developed a method for the facial expression recognition of a speaker. For facial expression recognition, we selected three static images at the timing positions of just before speaking and while speaking the phonemes of the first and last vowels. Then, only the static image of the front-view face was used for facial expression recognition. However, frequent updates of the training data were time-consuming. To reduce the time for updates, we found that the classifications of "neutral", "happy", and "others" were efficient and accurate for facial expression recognition. Using the proposed method with updated training data of "happy" and "neutral" after an interval such as approximately three and a half years, the facial expressions of two subjects were discriminable with 87.0 % accuracy for the facial expressions of "happy", "neutral", and "others" when exhibiting the intentional facial expressions of "angry", "happy", "neutral", "sad", and "surprised".

Robust Facial Expression Recognition via Compressive Sensing

Recently, compressive sensing (CS) has attracted increasing attention in the areas of signal processing, computer vision and pattern recognition. In this paper, a new method based on the CS theory is presented for robust facial expression recognition. The CS theory is used to construct a sparse representation classifier (SRC). The effectiveness and robustness of the SRC method is investigated on clean and occluded facial expression images. Three typical facial features, i.e., the raw pixels, Gabor wavelets representation and local binary patterns (LBP), are extracted to evaluate the performance of the SRC method. Compared with the nearest neighbor (NN), linear support vector machines (SVM) and the nearest subspace (NS), experimental results on the popular Cohn-Kanade facial expression database demonstrate that the SRC method obtains better performance and stronger robustness to corruption and occlusion on robust facial expression recognition tasks.

Robust Facial Expression Recognition Based on Local Directional Pattern

Automatic facial expression recognition has many potential applications in different areas of human computer interaction. However, they are not yet fully realized due to the lack of an effective facial feature descriptor. In this paper, we present a new appearance-based feature descriptor, the local directional pattern (LDP), to represent facial geometry and analyze its performance in expression recognition. An LDP feature is obtained by computing the edge response values in 8 directions at each pixel and encoding them into an 8 bit binary number using the relative strength of these edge responses. The LDP descriptor, a distribution of LDP codes within an image or image patch, is used to describe each expression image. The effectiveness of dimensionality reduction techniques, such as principal component analysis and AdaBoost, is also analyzed in terms of computational cost saving and classification accuracy. Two well-known machine learning methods, template matching and support vector machine, are used for classification using the Cohn-Kanade and Japanese female facial expression databases. Better classification accuracy shows the superiority of LDP descriptor against other appearance-based feature descriptors.

表情の左右対称性に注目した簡易な表情認識手法に関する実験的検討

This study aims to construct a consistent system from a position sensing of face to the facial expression recognition for the realization of comfortable man-machine communication. In this article, we examine the robust facial expression recognition method which is using the feature points. The feature points are defined as its can be used on the face images which sizes are different by the distance from the camera. The facial expression symmetry has the possibility to reduce the feature points which is used for the facial expression recognition. And it enables the expression recognition without limiting the direction of face.

Subject independent facial expression recognition with robust face detection using a convolutional neural network

Reliable detection of ordinary facial expressions (e.g. smile) despite the variability among individuals as well as face appearance is an important step toward the realization of perceptual user interface with autonomous perception of persons. We describe a rule-based algorithm for robust facial expression recognition combined with robust face detection using a convolutional neural network. In this study, we address the problem of subject independence as well as translation, rotation, and scale invariance in the recognition of facial expression. The result shows reliable detection of smiles with recognition rate of 97.6% for 5600 still images of more than 10 subjects. The proposed algorithm demonstrated the ability to discriminate smiling from talking based on the saliency score obtained from voting visual cues. To the best of our knowledge, it is the first facial expression recognition model with the property of subject independence combined with robustness to variability in facial appearance.