Facial Micro-expression Recognition Research Articles

OFF-ApexNet on micro-expression recognition system

When a person attempts to conceal an emotion, the genuine emotion is manifest as a micro-expression. Exploration of automatic facial micro-expression recognition systems is relatively new in the computer vision domain. This is due to the difficulty in implementing optimal feature extraction methods to cope with the subtlety and brief motion characteristics of the expression. Most of the existing approaches extract the subtle facial movements based on hand-crafted features. In this paper, we address the micro-expression recognition task with a convolutional neural network (CNN) architecture, which well integrates the features extracted from each video. We introduce the Optical Flow Features from Apex frame Network (OFF-ApexNet). This is a new feature descriptor that combines the optical flow guided context with the CNN. Firstly, we obtain the location of the apex frame from each video sequence as it portrays the highest intensity of facial motion among all frames. Then, the optical flow information are attained from the apex frame and a reference frame (i.e., onset frame). Finally, the optical flow features are fed into a pre-designed CNN model for further feature enhancement as well as to carry out the expression classification. To evaluate the effectiveness of OFF-ApexNet method, comprehensive evaluations are conducted on three public spontaneous micro-expression datasets (i.e., SMIC, CASME II and SAMM). The promising recognition result suggests that the proposed method can optimally describe the significant micro-expression details. In particular, we report that, in a multi-database with leave-one-subject-out cross-validation (LOSOCV) experimental protocol, the recognition performance reaches 74.60% of recognition accuracy and F-measure of 71.04%. We also note that this is the first work that performs cross-dataset validation on three databases in this domain.

Discriminative Spatiotemporal Local Binary Pattern with Revisited Integral Projection for Spontaneous Facial Micro-Expression Recognition

Recently, there have been increasing interests in inferring mirco-expression from facial image sequences. Due to subtle facial movement of micro-expressions, feature extraction has become an important and critical issue for spontaneous facial micro-expression recognition. Recent works used spatiotemporal local binary pattern (STLBP) for micro-expression recognition and considered dynamic texture information to represent face images. However, they miss the shape attribute of face images. On the other hand, they extract the spatiotemporal features from the global face regions while ignore the discriminative information between two micro-expression classes. The above-mentioned problems seriously limit the application of STLBP to micro-expression recognition. In this paper, we propose a discriminative spatiotemporal local binary pattern based on an integral projection to resolve the problems of STLBP for micro-expression recognition. First, we revisit an integral projection for preserving the shape attribute of micro-expressions by using robust principal component analysis. Furthermore, a revisited integral projection is incorporated with local binary pattern across spatial and temporal domains. Specifically, we extract the novel spatiotemporal features incorporating shape attributes into spatiotemporal texture features. For increasing the discrimination of micro-expressions, we propose a new feature selection based on Laplacian method to extract the discriminative information for facial micro-expression recognition. Intensive experiments are conducted on three availably published micro-expression databases including CASME, CASME2 and SMIC databases. We compare our method with the state-of-the-art algorithms. Experimental results demonstrate that our proposed method achieves promising performance for micro-expression recognition.

Extended Local Binary Patterns for Efficient and Robust Spontaneous Facial Micro-Expression Recognition

Facial Micro-Expressions (MEs) are spontaneous, involuntary facial movements when a person experiences an emotion but deliberately or unconsciously attempts to conceal his or her genuine emotions. Recently, ME recognition has attracted increasing attention due to its potential applications such as clinical diagnosis, business negotiation, interrogations, and security. However, it is expensive to build large scale ME datasets, mainly due to the difficulty of inducing spontaneous MEs. This limits the application of deep learning techniques which require lots of training data. In this paper, we propose a simple, efficient yet robust descriptor called Extended Local Binary Patterns on Three Orthogonal Planes (ELBPTOP) for ME recognition. ELBPTOP consists of three complementary binary descriptors: LBPTOP and two novel ones Radial Difference LBPTOP (RDLBPTOP) and Angular Difference LBPTOP (ADLBPTOP), which explore the local second order information along the radial and angular directions contained in ME video sequences. ELBPTOP is a novel ME descriptor inspired by unique and subtle facial movements. It is computationally efficient and only marginally increases the cost of computing LBPTOP, yet is extremely effective for ME recognition. In addition, by firstly introducing Whitened Principal Component Analysis (WPCA) to ME recognition, we can further obtain more compact and discriminative feature representations, then achieve significantly computational savings. Extensive experimental evaluation on three popular spontaneous ME datasets SMIC, CASME II and SAMM show that our proposed ELBPTOP approach significantly outperforms the previous state-of-the-art on all three single evaluated datasets and achieves promising results on cross-database recognition.Our code will be made available.

Facial micro-expression recognition based on the fusion of deep learning and enhanced optical flow

Micro-expression is a kind of split-second subtle expression which could not be controlled by the autonomic nervous system. Micro-expression indicates that a person is hiding his truly emotion consciously. Because the micro-expression is closely interrelated with lie detection, micro-expression recognition has various potential applications in many domains, such as the public security, the clinical medicine, the investigation and the interrogation. Because recognizing the micro-expression through human observation is very difficult, researchers focus on the automatic micro-expression recognition. This research proposed a novel algorithm for automatic micro-expression recognition which combined a deep multi-task convolutional network for detecting the facial landmarks and a fused deep convolutional network for estimating the optical flow features of the micro-expression. Firstly, this research employed the deep multi-task convolutional network to detect facial landmarks with the manifold related tasks and divided the facial region by utilizing these facial landmarks. Furthermore, a fused convolutional network was applied for extracting the optical flow features from the facial regions which contain the muscle changes when the micro-expression presents. Finally the enhanced optical flow was applied for refining the information of the features and these refined optical flow features were classified by Support Vector Machine classifier for recognizing the micro-expression. The result of experiments on two spontaneous micro-expression database demonstrated that the method proposed in this paper achieved good performance in micro-expression recognition.

A survey: facial micro-expression recognition

Facial expression recognition plays a crucial role in a wide range of applications of psychotherapy, security systems, marketing, commerce and much more. Detecting a macro-expression, which is a direct representation of an ‘emotion,’ is a relatively straight-forward task. Playing a pivotal role as macro-expressions, micro-expressions are more accurate indicators of a train of thought or even subtle, passive or involuntary thoughts. Compared to macro-expressions, identifying micro-expressions is a much more challenging research question because their time spans are narrowed down to a fraction of a second, and can only be defined using a broader classification scale. This paper is an all-inclusive survey-cum-analysis of the various micro-expression recognition techniques. We analyze the general framework for micro-expression recognition system by decomposing the pipeline into fundamental components, namely face detecting, pre-processing, facial feature detection and extraction, datasets, and classification. We discuss the role of these elements and highlight the models and new trends that are followed in their design. Moreover, we provide an extensive analysis of micro-expression recognition systems by comparing their performance. We also discuss the new deep learning features that can, in the near future, replace the hand-crafted features for facial micro-expression recognition. This survey has been developed, focusing on the methodologies applied, databases used, performance regarding recognition accuracy and comparing these to distil the gaps in the efficiencies, future scope, and research potentials. Through this survey, we intend to look into this problem and develop a comprehensive and efficient recognition scheme. This study allows us to identify open issues and to determine future directions for designing real-world micro-expression recognition systems.

Effective recognition of facial micro-expressions with video motion magnification

Facial expression recognition has been intensively studied for decades, notably by the psychology community and more recently the pattern recognition community. What is more challenging, and the subject of more recent research, is the problem of recognizing subtle emotions exhibited by so-called micro-expressions. Recognizing a micro-expression is substantially more challenging than conventional expression recognition because these micro-expressions are only temporally exhibited in a fraction of a second and involve minute spatial changes. Until now, work in this field is at a nascent stage, with only a few existing micro-expression databases and methods. In this article, we propose a new micro-expression recognition approach based on the Eulerian motion magnification technique, which could reveal the hidden information and accentuate the subtle changes in micro-expression motion. Validation of our proposal was done on the recently proposed CASME II dataset in comparison with baseline and state-of-the-art methods. We achieve a good recognition accuracy of up to 75.30 % by using leave-one-out cross validation evaluation protocol. Extensive experiments on various factors at play further demonstrate the effectiveness of our proposed approach.

A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition

Micro-expressions are brief facial movements characterized by short duration, involuntariness and low intensity. Recognition of spontaneous facial micro-expressions is a great challenge. In this paper, we propose a simple yet effective Main Directional Mean Optical-flow (MDMO) feature for micro-expression recognition. We apply a robust optical flow method on micro-expression video clips and partition the facial area into regions of interest (ROIs) based partially on action units. The MDMO is a ROI-based, normalized statistic feature that considers both local statistic motion information and its spatial location. One of the significant characteristics of MDMO is that its feature dimension is small. The length of a MDMO feature vector is $36 \times 2=72$ , where $36$ is the number of ROIs. Furthermore, to reduce the influence of noise due to head movements, we propose an optical-flow-driven method to align all frames of a micro-expression video clip. Finally, a SVM classifier with the proposed MDMO feature is adopted for micro-expression recognition. Experimental results on three spontaneous micro-expression databases, namely SMIC, CASME and CASME II, show that the MDMO can achieve better performance than two state-of-the-art baseline features, i.e., LBP-TOP and HOOF.

An Analysis of Perspectives for Using High-Speed Cameras in Processing Dynamic Video Information

In this paper, we review the actual and perspective areas of use of high-speed video cameras. We discuss the possibility of applying high-speed cameras in the field of human-computer interaction to detect dynamic video information (including visual speech). We also describe main tasks, which can be solved with high-speed cameras, such as: automatic lip-reading, eye blink detection, facial micro-expression recognition, etc. We identify potential challenges associated with the introduction of high-speed video cameras and analyze the conditions of research area. Besides, we analyze state-of-the-art in the field at the moment and prove that there is an urgent need for further scientific and technical developments in this area. We propose some advanced applications and tasks in the human-computer interaction domain, where high-speed video capturing can be useful, such as audio-visual continuous speech recognition and automatic reading speech by lips. In further research, we will implement such a multimodal system for audio-visual Russian speech recognition using a microphone and a high-speed video camera JAI Pulnix.

Micro-expression recognition training in medical students: a pilot study

BackgroundPatients provide emotional cues during consultations which may be verbal or non-verbal. Many studies focus on patient verbal cues as predictors of physicians' ability to recognize and address patient needs but this project focused on non-verbal cues in the form of facial micro-expressions. This pilot study investigated first year medical students' (n = 75) identified as being either good or poor communicators abilities to detect emotional micro-expressions before and after training using the Micro Expression Training Tool (METT) .MethodsThe sample consisted of 24 first year medical students, 9 were from the lowest performance quartile in a communication skills OSCE (Objective Structured Clinical Exam) station and 15 were from the highest performance quartile. These students completed the METT individually, recording pre- and post-assessment scores. Students were also invited to provide their views on the training.ResultsNo difference in pre-assessment scores was found between the lowest and highest quartile groups (P = 0.797). After training, students in the high quartile showed significant improvement in the recognition of facial micro-expressions (P = 0.014). The lowest quartile students showed no improvement (P = 0.799).ConclusionIn conclusion, this pilot study showed there was no difference between the ability of medical undergraduate students assessed as being good communicators and those assessed as poor communicators to identify facial micro-expressions. But, the study did highlight that those students demonstrating good general clinical communication benefited from the training aspect of the METT, whereas low performing students did not gain. Why this should be the case is not clear and further investigation should be carried out to determine why lowest quartile students did not benefit.