Facial Micro-expression Recognition Research Articles

Facial micro-expression recognition using stochastic graph convolutional network and dual transferred learning

Micro-expression recognition (MER) has drawn increasing attention due to its wide application in lie detection, criminal detection and psychological consultation. However, the best recognition accuracy on recent public dataset is still low compared to the accuracy of macro-expression recognition. In this paper, we propose a novel graph convolution network (GCN) for MER achieving state-of-the-art accuracy. Different to existing GCN with fixed graph structure, we define a stochastic graph structure in which some neighbors are selected randomly. As shown by numerical examples, randomness enables better feature characterization while reducing computational complexity. The whole network consists of two branches, one is the spatial branch taking micro-expression images as input, the other is the temporal branch taking optical flow images as input. Because the micro-expression dataset does not have enough images for training the GCN, we employ the transfer learning mechanism. That is, different stochastic GCNs (SGCN) have been trained by the macro-expression dataset in the source network. Then the well-trained SGCNs are transferred to the target network. It is shown that our proposed method achieves the state-of-art performance on all four well-known datasets. This paper explores stochastic GCN and transfer learning with this random structure in the MER task, which is of great importance to improve the recognition performance.

Athlete facial micro-expression recognition method based on graph convolutional neural network

Athlete facial micro-expression recognition method based on graph convolutional neural network

Facial micro-expression recognition method based on CNN and Transformer mixed model

Facial micro-expression recognition method based on CNN and Transformer mixed model

Trainee psychotherapists’ emotion recognition accuracy during 1.5 years of psychotherapy education compared to a control group: no improvement after psychotherapy training

The ability to recognize and work with patients’ emotions is considered an important part of most psychotherapy approaches. Surprisingly, there is little systematic research on psychotherapists’ ability to recognize other people’s emotional expressions. In this study, we compared trainee psychotherapists’ nonverbal emotion recognition accuracy to a control group of undergraduate students at two time points: at the beginning and at the end of one and a half years of theoretical and practical psychotherapy training. Emotion recognition accuracy (ERA) was assessed using two standardized computer tasks, one for recognition of dynamic multimodal (facial, bodily, vocal) expressions and one for recognition of facial micro expressions. Initially, 154 participants enrolled in the study, 72 also took part in the follow-up. The trainee psychotherapists were moderately better at recognizing multimodal expressions, and slightly better at recognizing facial micro expressions, than the control group at the first test occasion. However, mixed multilevel modeling indicated that the ERA change trajectories for the two groups differed significantly. While the control group improved in their ability to recognize multimodal emotional expressions from pretest to follow-up, the trainee psychotherapists did not. Both groups improved their micro expression recognition accuracy, but the slope for the control group was significantly steeper than the trainee psychotherapists’. These results suggest that psychotherapy education and clinical training do not always contribute to improved emotion recognition accuracy beyond what could be expected due to time or other factors. Possible reasons for that finding as well as implications for the psychotherapy education are discussed.

Self-supervised Learning for Expression Recognition on Small-scale Data Set

In this paper, we propose a facial micro-expression recognition method utilizing self-supervised learning and Vision Transformer. We employ a contrast learning approach for self-supervision and extract image features using an attention mechanism. Various data augmentation techniques were utilized, and we specifically designed an enhancement method for facial recognition. By combining the strengths of the Vision Transformer and CNN models for feature extraction, our approach achieves improved recognition accuracy, even with limited labeled data. Experimental evaluation shows that our proposed method has good results in facial micro-expression recognition tasks.

Identification of micro expressions in a video sequence by Euclidean distance of the facial contours

This paper presents an automatic facial micro-expression recognition system (FMER) from video sequence. Identification and classification are performed on basic expressions: happy, surprise, fear, disgust, sadness, anger, and neutral states. The system integrates three main steps. The first step consists in face detection and tracking over three consecutive frames. In the second step, the facial contour extraction is performed on each frame to build Euclidean distance maps. The last task corresponds to the classification which is achieved with two methods; the SVM and using convolutional neural networks. Experimental evaluation of the proposed system for facial micro-expression identification is performed on the well-known databases (Chon and Kanade and CASME II), with six and seven facial expressions for each classification method.

Multimodal Attention Dynamic Fusion Network for Facial Micro-Expression Recognition.

The emotional changes in facial micro-expressions are combinations of action units. The researchers have revealed that action units can be used as additional auxiliary data to improve facial micro-expression recognition. Most of the researchers attempt to fuse image features and action unit information. However, these works ignore the impact of action units on the facial image feature extraction process. Therefore, this paper proposes a local detail feature enhancement model based on a multimodal dynamic attention fusion network (MADFN) method for micro-expression recognition. This method uses a masked autoencoder based on learnable class tokens to remove local areas with low emotional expression ability in micro-expression images. Then, we utilize the action unit dynamic fusion module to fuse action unit representation to improve the potential representation ability of image features. The state-of-the-art performance of our proposed model is evaluated and verified on SMIC, CASME II, SAMM, and their combined 3DB-Combined datasets. The experimental results demonstrated that the proposed model achieved competitive performance with accuracy rates of 81.71%, 82.11%, and 77.21% on SMIC, CASME II, and SAMM datasets, respectively, that show the MADFN model can help to improve the discrimination of facial image emotional features.

Multi-scale fusion visual attention network for facial micro-expression recognition.

Micro-expressions are facial muscle movements that hide genuine emotions. In response to the challenge of micro-expression low-intensity, recent studies have attempted to locate localized areas of facial muscle movement. However, this ignores the feature redundancy caused by the inaccurate locating of the regions of interest. This paper proposes a novel multi-scale fusion visual attention network (MFVAN), which learns multi-scale local attention weights to mask regions of redundancy features. Specifically, this model extracts the multi-scale features of the apex frame in the micro-expression video clips by convolutional neural networks. The attention mechanism focuses on the weights of local region features in the multi-scale feature maps. Then, we mask operate redundancy regions in multi-scale features and fuse local features with high attention weights for micro-expression recognition. The self-supervision and transfer learning reduce the influence of individual identity attributes and increase the robustness of multi-scale feature maps. Finally, the multi-scale classification loss, mask loss, and removing individual identity attributes loss joint to optimize the model. The proposed MFVAN method is evaluated on SMIC, CASME II, SAMM, and 3DB-Combined datasets that achieve state-of-the-art performance. The experimental results show that focusing on local at the multi-scale contributes to micro-expression recognition. This paper proposed MFVAN model is the first to combine image generation with visual attention mechanisms to solve the combination challenge problem of individual identity attribute interference and low-intensity facial muscle movements. Meanwhile, the MFVAN model reveal the impact of individual attributes on the localization of local ROIs. The experimental results show that a multi-scale fusion visual attention network contributes to micro-expression recognition.

Sensitivity and response criterion in facial micro-expression recognition among internet gaming disorder

Sensitivity and response criterion in facial micro-expression recognition among internet gaming disorder

Diverse local facial behaviors learning from enhanced expression flow for microexpression recognition

Facial microexpression recognition (FMER) has recently gained increasing attention in the interrogation and clinical diagnosis fields. However, accurate FMER suffers from not only the low-intensity muscle movements and a short duration of microexpressions but also limited training data. To address the abovementioned issues, this paper employs facial u/v/s images to explore short durationlow-intensity muscle movements, which can be extracted from the expression flow. Specifically, we propose a motion detail enhancement method to disentangle microexpression-irrelevant motions from the expression flow. Then, a novel data augmentation strategy based on different u-v weights is introduced to generate diverse s components, resolving the limited scale issue. Afterward, the enhanced facial u/v/s images are fed to a novel local-diverse facial microexpression recognition network (LD-FMERN) for microexpression-related feature extraction, where a spatial-channel modulator is used to refine the extracted features. A locally diverse feature mining strategy further enhances the refined features, forcing the network to focus on small and diverse facial regions. Following the law of human vision, we propose an adaptive loss function, which comprises a supervised cross-entropy loss and a self-supervised local-diverse loss, to optimize the network. Extensive quantitative and qualitative evaluations on benchmark datasets, CASMEII, SAMM, and MMEW, demonstrate the improvements. Comparisons with state-of-the-art FMER methods reveal the superiority of the proposed method.

Facial Micro-Expression Recognition Enhanced by Score Fusion and a Hybrid Model from Convolutional LSTM and Vision Transformer

In the billions of faces that are shaped by thousands of different cultures and ethnicities, one thing remains universal: the way emotions are expressed. To take the next step in human–machine interactions, a machine (e.g., a humanoid robot) must be able to clarify facial emotions. Allowing systems to recognize micro-expressions affords the machine a deeper dive into a person’s true feelings, which will take human emotion into account while making optimal decisions. For instance, these machines will be able to detect dangerous situations, alert caregivers to challenges, and provide appropriate responses. Micro-expressions are involuntary and transient facial expressions capable of revealing genuine emotions. We propose a new hybrid neural network (NN) model capable of micro-expression recognition in real-time applications. Several NN models are first compared in this study. Then, a hybrid NN model is created by combining a convolutional neural network (CNN), a recurrent neural network (RNN, e.g., long short-term memory (LSTM)), and a vision transformer. The CNN can extract spatial features (within a neighborhood of an image), whereas the LSTM can summarize temporal features. In addition, a transformer with an attention mechanism can capture sparse spatial relations residing in an image or between frames in a video clip. The inputs of the model are short facial videos, while the outputs are the micro-expressions recognized from the videos. The NN models are trained and tested with publicly available facial micro-expression datasets to recognize different micro-expressions (e.g., happiness, fear, anger, surprise, disgust, sadness). Score fusion and improvement metrics are also presented in our experiments. The results of our proposed models are compared with that of literature-reported methods tested on the same datasets. The proposed hybrid model performs the best, where score fusion can dramatically increase recognition performance.

C3DBed: Facial micro-expression recognition with three-dimensional convolutional neural network embedding in transformer model

Facial micro-expression is often used for emotional recognition of people in a high-risk or pressure scene, which may reflect genuine emotions due to the low intensity of facial action units. Current methods focus on locating regions with emotional changes and cropping these regions for local feature extraction. However, these methods may lead to the problem of information redundancy caused by overlapping cropped regions. This paper proposes a novel three-dimensional convolutional neural network embedding in the transformer model (C3DBed). This model learns the attention weight of each local region of the micro-expression image, thereby perceiving the detail changes of the facial image and extracting robust local detail features. Solve the problem of model complexity and information redundancy caused by low-intensity local area positioning of facial muscle movement. The experiment results demonstrated that the proposed C3DBed model achieved competitive performance with accuracy rates of 78.04%, 77.64%, and 75.73% on SMIC, CASME II, and SAMM datasets, respectively.

MERCoL: video-based facial micro-expression recognition via bimodal contrastive learning

MERCoL: video-based facial micro-expression recognition via bimodal contrastive learning

Multi-Stream Deep Convolution Neural Network With Ensemble Learning for Facial Micro-Expression Recognition

Multi-Stream Deep Convolution Neural Network With Ensemble Learning for Facial Micro-Expression Recognition

MERCoL: video-based facial micro-expression recognition via bimodal contrastive learning

MERCoL: video-based facial micro-expression recognition via bimodal contrastive learning

Facial Micro Expression Recognition for Feature Point Tracking using Apex Frames on CASME II Database

Micro-expressions are facial expressions that occur inadvertently to hide true feelings (emotional leaks). Although previous studies used the entire face area and all frames in the video dataset, this resulted in relatively long computation time and data redundancy. The main contribution of this research is to apply recognition micro-expression analysis using a comparison of apex frames with manual (handcrafted) and random sampling of frames and applying feature point tracking to the brow area and corners of the lips. The method for forming feature points in the facial area uses Discriminative Response Map Fitting (DRMF), then facial feature points are tracked using Kanade-Lucas-Tomasi (KLT). This feature point tracking produces motion feature data as feature extraction data. Finally, a comparative analysis of the classification method using the Support Vector Machine (SVM) and MLP-Backpropagation was conducted using the CASME II dataset. The experimental results of this study show significant results with an accuracy of 81.3% on MLP-Backpropagation and an average computing time of 1.45 seconds for each video. From the results of this study, information on the apex phase can contribute information that is very important for facial micro-expression recognition.

Thermoacoustic environment comfort evaluation method based on facial micro-expression recognition

At present, subjective questionnaire surveys and physiological parameters measured are mainly used to collect the comprehensive response of humans to the environment, evaluating the environmental comfort subjectively and objectively. However, some measuring instruments make people feel uncomfortable and interfere with the natural state of the human body. Some measuring instruments are too bulky to carry. In addition, the changes in physiological parameters are not synchronized with the changes in human feeling, and there is a lag phenomenon, which cannot immediately reflect the human body current feeling. This study considers that the facial expression is an expression form of human emotion, psychology, and brain activities. When touch, hearing, vision, taste, and smell are stimulated by external factors, the facial expression will change. In view of this, this study examines the thermoacoustic environmental comfort based on facial micro-expression recognition. Firstly, the facial micro-expression database based on environment comfort (FMEEC) is constructed. Secondly, a convolutional neural network (CNN) is used to build the micro-expression recognition model (MERCNN). Finally, the MERCNN model is trained to converge with training and verification samples. The micro-expression images collected in the artificial climate chamber and classroom environment are used to test the model's performance. The evaluation results of the MERCNN are compared with the results of subjective questionnaires and the predicted mean vote (PMV) model. The results show that the MERCNN can accurately evaluate the thermoacoustic environment comfort through face micro-expression recognition.

Facial micro-expression recognition impairment and its relationship with social anxiety in internet gaming disorder

Facial micro-expression recognition impairment and its relationship with social anxiety in internet gaming disorder

Motion magnification multi-feature relation network for facial microexpression recognition

Microexpressions cannot be observed easily due to their short duration and small-expression range. These properties pose considerable challenges for the recognition of microexpressions. Thus, video motion magnification techniques help us to see small motions previously invisible to the naked eye. This study aimed to enhance the microexpression features with the help of motion amplification technology. Also, a motion magnification multi-feature relation network (MMFRN) combining video motion amplification and two feature relation modules was proposed. The spatial feature is enlarged while completing the spatial feature extraction, which is used for classification. In addition, we transferred Resnet50 network to extract the global features and improve feature comprehensiveness. The magnification of the features is controlled through hyperparameter amplification factor α. The effects of different magnification factors on the results are compared, and the best is selected. The experiments have verified that the enlarged network can resolve the misclassification problem caused by the one-to-one correspondence between microexpressions and facial action coding units. On CASME II datasets, MMFRN outperforms the traditional recognition methods and other neural networks.

Late Fusion-Based Video Transformer for Facial Micro-Expression Recognition

In this article, we propose a novel model for facial micro-expression (FME) recognition. The proposed model basically comprises a transformer, which is recently used for computer vision and has never been used for FME recognition. A transformer requires a huge amount of data compared to a convolution neural network. Then, we use motion features, such as optical flow and late fusion to complement the lack of FME dataset. The proposed method was verified and evaluated using the SMIC and CASME II datasets. Our approach achieved state-of-the-art (SOTA) performance of 0.7447 and 73.17% in SMIC in terms of unweighted F1 score (UF1) and accuracy (Acc.), respectively, which are 0.31 and 1.8% higher than previous SOTA. Furthermore, UF1 of 0.7106 and Acc. of 70.68% were shown in the CASME II experiment, which are comparable with SOTA.