Micro-expression Recognition Research Articles

A Review of Automated Micro-expression Analysis

Micro-expression is a type of facial expression that is manifested for a very short duration. It is difficult to recognize the expression manually because it involves very subtle facial movements. Such expressions often occur unconsciously, and therefore are defined as a basis to help identify the real human emotions. Hence, an automated approach to micro-expression recognition has become a popular research topic of interest recently. Historically, the early researches on automated micro-expression have utilized traditional machine learning methods, while the more recent development has focused on the deep learning approach. Compared to traditional machine learning, which relies on manual feature processing and requires the use of formulated rules, deep learning networks produce more accurate micro-expression recognition performances through an end-to-end methodology, whereby the features of interest were extracted optimally through the training process, utilizing a large set of data. This paper reviews the developments and trends in micro-expression recognition from the earlier studies (hand-crafted approach) to the present studies (deep learning approach). Some of the important topics that will be covered include the detection of micro-expression from short videos, apex frame spotting, micro-expression recognition as well as performance discussion on the reviewed methods. Furthermore, major limitations that hamper the development of automated micro-expression recognition systems are also analyzed, followed by recommendations of possible future research directions.

The effect of facial attractiveness on micro-expression recognition.

Micro-expression (ME) is an extremely quick and uncontrollable facial movement that lasts for 40–200 ms and reveals thoughts and feelings that an individual attempts to cover up. Though much more difficult to detect and recognize, ME recognition is similar to macro-expression recognition in that it is influenced by facial features. Previous studies suggested that facial attractiveness could influence facial expression recognition processing. However, it remains unclear whether facial attractiveness could also influence ME recognition. Addressing this issue, this study tested 38 participants with two ME recognition tasks in a static condition or dynamically. Three different MEs (positive, neutral, and negative) at two attractiveness levels (attractive, unattractive). The results showed that participants recognized MEs on attractive faces much quicker than on unattractive ones, and there was a significant interaction between ME and facial attractiveness. Furthermore, attractive happy faces were recognized faster in both the static and the dynamic conditions, highlighting the happiness superiority effect. Therefore, our results provided the first evidence that facial attractiveness could influence ME recognition in a static condition or dynamically.

Differences in brain activations between micro- and macro-expressions based on electroencephalography.

Micro-expressions can reflect an individual’s subjective emotions and true mental state and are widely used in the fields of mental health, justice, law enforcement, intelligence, and security. However, the current approach based on image and expert assessment-based micro-expression recognition technology has limitations such as limited application scenarios and time consumption. Therefore, to overcome these limitations, this study is the first to explore the brain mechanisms of micro-expressions and their differences from macro-expressions from a neuroscientific perspective. This can be a foundation for micro-expression recognition based on EEG signals. We designed a real-time supervision and emotional expression suppression (SEES) experimental paradigm to synchronously collect facial expressions and electroencephalograms. Electroencephalogram signals were analyzed at the scalp and source levels to determine the temporal and spatial neural patterns of micro- and macro-expressions. We found that micro-expressions were more strongly activated in the premotor cortex, supplementary motor cortex, and middle frontal gyrus in frontal regions under positive emotions than macro-expressions. Under negative emotions, micro-expressions were more weakly activated in the somatosensory cortex and corneal gyrus regions than macro-expressions. The activation of the right temporoparietal junction (rTPJ) was stronger in micro-expressions under positive than negative emotions. The reason for this difference is that the pathways of facial control are different; the production of micro-expressions under positive emotion is dependent on the control of the face, while micro-expressions under negative emotions are more dependent on the intensity of the emotion.

Micro-Expression Recognition Using Uncertainty-Aware Magnification-Robust Networks.

A micro-expression (ME) is a kind of involuntary facial expressions, which commonly occurs with subtle intensity. The accurately recognition ME, a. k. a. micro-expression recognition (MER), has a number of potential applications, e.g., interrogation and clinical diagnosis. Therefore, the subject has received a high level of attention among researchers in affective computing and pattern recognition communities. In this paper, we proposed a straightforward and effective deep learning method called uncertainty-aware magnification-robust networks (UAMRN) for MER, which attempts to address two key issues in MER including the low intensity of ME and imbalance of ME samples. Specifically, to better distinguish subtle ME movements, we reconstructed a new sequence by magnifying the ME intensity. Furthermore, a sparse self-attention (SSA) block was implemented which rectifies the standard self-attention with locality sensitive hashing (LSH), resulting in the suppression of artefacts generated during magnification. On the other hand, for the class imbalance problem, we guided the network optimization based on the confidence about the estimation, through which the samples from rare classes were allotted greater uncertainty and thus trained more carefully. We conducted the experiments on three public ME databases, i.e., CASME II, SAMM and SMIC-HS, the results of which demonstrate improvement compared to recent state-of-the-art MER methods.

Improved Convolutional Neural Networks for Course Teaching Quality Assessment

The cultivation of innovative talents is closely related to the quality of course teaching, and there is a correlation between facial expressions and the effectiveness of classroom teaching. In this paper, a separate long-term recursive convolutional network (SLRCN) microexpression recognition algorithm is proposed using deep learning technology for building a course teaching effectiveness evaluation model. Firstly, facial image sequences are extracted from microexpression data sets, and the transfer learning method is introduced to extract spatial features of facial expression frames through pretrained convolutional neural network model to reduce the risk of overfitting in network training. The extracted features of video sequences were input into long short-team memory (LSTM) to process temporal features. Experimental results show that SLRCN algorithm has the best performance in training set and test set. It has the best performance in ROC curve. This effectively distinguishes between seven different expressions in the database. The model proposed in this paper can obtain the changes of students’ facial expressions in class and evaluate students’ learning status, thus promoting the improvement of teaching quality. It provides a new method of course teaching quality evaluation.

Transcranial Direct Current Stimulation Over the Right Temporal Parietal Junction Facilitates Spontaneous Micro-Expression Recognition.

Micro-expressions are fleeting and subtle emotional expressions. As they are spontaneous and uncontrollable by one’s mind, micro-expressions are considered an indicator of genuine emotions. Their accurate recognition and interpretation promote interpersonal interaction and social communication. Therefore, enhancing the ability to recognize micro-expressions has captured much attention. In the current study, we investigated the effects of training on micro-expression recognition with a Chinese version of the Micro-Expression Training Tool (METT). Our goal was to confirm whether the recognition accuracy of spontaneous micro-expressions could be improved through training and brain stimulation. Since the right temporal parietal junction (rTPJ) has been shown to be involved in the explicit process of facial emotion recognition, we hypothesized that the rTPJ would play a role in facilitating the recognition of micro-expressions. The results showed that anodal transcranial direct-current stimulation (tDCS) of the rTPJ indeed improved the recognition of spontaneous micro-expressions, especially for those associated with fear. The improved accuracy of recognizing fear spontaneous micro-expressions was positively correlated with personal distress in the anodal group but not in the sham group. Our study supports that the combined use of tDCS and METT can be a viable way to train and enhance micro-expression recognition.

Micro-Expression Recognition based on the Multi-Color ULBP and Histogram of Gradient Direction from Six Intersection Planes

Micro-Expression Recognition based on the Multi-Color ULBP and Histogram of Gradient Direction from Six Intersection Planes

Lightweight ViT Model for Micro-Expression Recognition Enhanced by Transfer Learning.

As opposed to macro-expressions, micro-expressions are subtle and not easily detectable emotional expressions, often containing rich information about mental activities. The practical recognition of micro-expressions is essential in interrogation and healthcare. Neural networks are currently one of the most common approaches to micro-expression recognition. Still, neural networks often increase their complexity when improving accuracy, and overly large neural networks require extremely high hardware requirements for running equipment. In recent years, vision transformers based on self-attentive mechanisms have achieved accuracy in image recognition and classification that is no less than that of neural networks. Still, the drawback is that without the image-specific biases inherent to neural networks, the cost of improving accuracy is an exponential increase in the number of parameters. This approach describes training a facial expression feature extractor by transfer learning and then fine-tuning and optimizing the MobileViT model to perform the micro-expression recognition task. First, the CASME II, SAMM, and SMIC datasets are combined into a compound dataset, and macro-expression samples are extracted from the three macro-expression datasets. Each macro-expression sample and micro-expression sample are pre-processed identically to make them similar. Second, the macro-expression samples were used to train the MobileNetV2 block in MobileViT as a facial expression feature extractor and to save the weights when the accuracy was highest. Finally, some of the hyperparameters of the MobileViT model are determined by grid search and then fed into the micro-expression samples for training. The samples are classified using an SVM classifier. In the experiments, the proposed method obtained an accuracy of 84.27%, and the time to process individual samples was only 35.4 ms. Comparative experiments show that the proposed method is comparable to state-of-the-art methods in terms of accuracy while improving recognition efficiency.

Oxytocin Impairs the Recognition of Micro-Expressions of Surprise and Disgust.

As fleeting facial expressions which reveal the emotion that a person tries to conceal, micro-expressions have great application potentials for fields like security, national defense and medical treatment. However, the physiological basis for the recognition of these facial expressions is poorly understood. In the present research, we utilized a double-blind, placebo-controlled, mixed-model experimental design to investigate the effects of oxytocin on the recognition of micro-expressions in three behavioral studies. Specifically, in Studies 1 and 2, participants were asked to perform a laboratory-based standardized micro-expression recognition task after self-administration of a single dose of intranasal oxytocin (40 IU) or placebo (containing all ingredients except for the neuropeptide). In Study 3, we further examined the effects of oxytocin on the recognition of natural micro-expressions. The results showed that intranasal oxytocin decreased the recognition speed for standardized intense micro-expressions of surprise (Study 1) and decreased the recognition accuracy for standardized subtle micro-expressions of disgust (Study 2). The results of Study 3 further revealed that intranasal oxytocin administration significantly reduced the recognition accuracy for natural micro-expressions of surprise and disgust. The present research is the first to investigate the effects of oxytocin on micro-expression recognition. It suggests that the oxytocin mainly plays an inhibiting role in the recognition of micro-expressions and there are fundamental differences in the neurophysiological basis for the recognition of micro-expressions and macro-expressions.

A novel micro-expression recognition algorithm using dual-stream combining optical flow and dynamic image convolutional neural networks

A novel micro-expression recognition algorithm using dual-stream combining optical flow and dynamic image convolutional neural networks

ME-PLAN: A deep prototypical learning with local attention network for dynamic micro-expression recognition

As one of the important psychological stress reactions, Micro-expressions (MEs) are spontaneous and subtle facial movements, which usually occur in a high-stake situation and can reveal genuine human feelings and cognition. ME, Recognition (MER) has essential applications in many fields such as lie detection, criminal investigation, and psychological healing. However, due to the challenges of learning discriminative ME features via fleeting facial subtle reactions as well as the shortage of available MEs data, this research topic is still far from well-studied. To this end, in this paper, we propose a deep prototypical learning framework, namely ME-PLAN, with a local attention mechanism for the MER problem. Specifically, ME-PLAN consists of two components, i.e., a 3D residual prototypical network and a local-wise attention module, where the former aims to learn the precise ME feature prototypes through expression-related knowledge transfer and episodic training, and the latter could facilitate the attention to the local facial movements. Furthermore, to alleviate the dilemma that most MER methods need to depend on manually annotated apex frames, we propose an apex frame spotting method with Unimodal Pattern Constrained (UPC) and further extract ME key-frames sequences based on the detected apex frames to train our proposed ME-PLAN in an end-to-end manner. Finally, through extensive experiments and interpretable analysis regarding the apex frame spotting and MER on composite-database, we demonstrate the superiority and effectiveness of the proposed methods.

Micro-expression recognition based on SqueezeNet and C3D

Micro-expression recognition based on SqueezeNet and C3D

Thermoacoustic environment comfort evaluation method based on facial micro-expression recognition

At present, subjective questionnaire surveys and physiological parameters measured are mainly used to collect the comprehensive response of humans to the environment, evaluating the environmental comfort subjectively and objectively. However, some measuring instruments make people feel uncomfortable and interfere with the natural state of the human body. Some measuring instruments are too bulky to carry. In addition, the changes in physiological parameters are not synchronized with the changes in human feeling, and there is a lag phenomenon, which cannot immediately reflect the human body current feeling. This study considers that the facial expression is an expression form of human emotion, psychology, and brain activities. When touch, hearing, vision, taste, and smell are stimulated by external factors, the facial expression will change. In view of this, this study examines the thermoacoustic environmental comfort based on facial micro-expression recognition. Firstly, the facial micro-expression database based on environment comfort (FMEEC) is constructed. Secondly, a convolutional neural network (CNN) is used to build the micro-expression recognition model (MERCNN). Finally, the MERCNN model is trained to converge with training and verification samples. The micro-expression images collected in the artificial climate chamber and classroom environment are used to test the model's performance. The evaluation results of the MERCNN are compared with the results of subjective questionnaires and the predicted mean vote (PMV) model. The results show that the MERCNN can accurately evaluate the thermoacoustic environment comfort through face micro-expression recognition.

Micro-Expression Recognition Based on Optical Flow and PCANet.

Micro-expressions are rapid and subtle facial movements. Different from ordinary facial expressions in our daily life, micro-expressions are very difficult to detect and recognize. In recent years, due to a wide range of potential applications in many domains, micro-expression recognition has aroused extensive attention from computer vision. Because available micro-expression datasets are very small, deep neural network models with a huge number of parameters are prone to over-fitting. In this article, we propose an OF-PCANet+ method for micro-expression recognition, in which we design a spatiotemporal feature learning strategy based on shallow PCANet+ model, and we incorporate optical flow sequence stacking with the PCANet+ network to learn discriminative spatiotemporal features. We conduct comprehensive experiments on publicly available SMIC and CASME2 datasets. The results show that our lightweight model obviously outperforms popular hand-crafted methods and also achieves comparable performances with deep learning based methods, such as 3D-FCNN and ELRCN.

Micro-expression recognition with attention mechanism and region enhancement

Micro-expression recognition with attention mechanism and region enhancement

The design of error-correcting output codes based deep forest for the micro-expression recognition

The design of error-correcting output codes based deep forest for the micro-expression recognition

Cross-database micro-expression recognition based on transfer double sparse learning

Cross-database micro-expression recognition based on transfer double sparse learning

Facial micro-expression recognition impairment and its relationship with social anxiety in internet gaming disorder

Facial micro-expression recognition impairment and its relationship with social anxiety in internet gaming disorder

Micro Expression Recognition Using Delaunay Triangulation and Voronoi Tessellation

Facial Expression Recognition is a visual cue used for conveying emotions and intentions between human beings. The micro-expressions (MEs) are not visible to the human eye, making it challenging to capture the minute changes in the facial areas as the expressions change. As a result, automating the detection of ME is a challenging task. This work utilizes Delaunay Triangulation and Voronoi Diagram properties to segment Region of Interest (ROI) based on Action Unit indexes. The ROI-based feature extraction aided in improving the performance of the Micro-Expression Recognition (MER) system. The Cross-Database Evaluation (CDE) and Holdout Database Evaluation (HDE) are performed on three publicly available datasets CASMEII, SAMM, and SMIC (HS). The proposed approach resulted in an improved Unweighted Average Recall (UAR) and Unweighted F1 (UF1) scores by 6.09% and 4.36%, respectively. The results obtained with CDE and HDE demonstrate that the proposed model is robust compared to earlier studies.

Comparative analysis of super-resolution reconstructed images for micro-expression recognition

It is an established fact that the genuineness of facial micro-expression is an effective means for estimating concealed emotions (Li et al. in Micro-expression recognition under low-resolution cases. SciTePress, Science and Technology Publications, Setúbal, 2019). Conventionally, analysis of these expressions has been performed using high resolution images which are ideal cases. However, in a real-world scenario, capturing expressions with high resolution images may not always be possible particularly using low-cost surveillance cameras. Faces captured using such cameras are often very tiny and of poor resolution. Due to the loss of discriminative features these images may not be of much use particularly for identifying certain minute facial details. To make these images useful, enhancing the textural information becomes essential and super-resolution algorithms can be ideal to achieve this. In this work, we utilize algorithms based on deep learning and generative adversarial network for transforming low-resolution micro-expression images into super-resolution images and examine their fitness particularly for micro-expression recognition. The proposed approach is tested on simulated dataset obtained from two popular spontaneous micro-expression datasets namely CASME II and SMIC-VIS; the experimental results demonstrate that the method achieved favourable results with the best recognition performance recorded as 61.63%. The significance of this work is: first, it thoroughly investigates reconstruction performance of several deep learning super-resolution algorithms on simulated low-quality micro-expression images; second, it provides a comprehensive analysis of the results obtained employing these reconstructed images to determine their contribution in addressing image quality issues specifically for micro-expression recognition.