Micro-expression Recognition Research Articles

Micro-expression recognition method integrating LBP and parallel attention mechanism

Aiming at the problems of weak intensity of facial micro-expression changes, background noise interference and low feature differentiation, a micro-expression recognition network integrating LBP and parallel attention mechanism is proposed. The network inputs RGB images into the densely connected improved Shuffle Stage branch to extract global facial features and enhance the association of contextual semantic information; the LBP image is input into the local texture feature branch composed of multi-scale hierarchical convolutional neural network to extract detail information; after the dual-branch feature extraction, the parallel attention mechanism is introduced at the back end of the network to improve the feature fusion ability, suppress background interference, and focus on the micro-expression feature interest area; the proposed method is tested on three public datasets including CASME, CASME II and SMIC, and the recognition accuracy rates are , and respectively ; the experimental results show that the proposed method effectively improves the accuracy of micro-expression recognition, which is better than many current advanced methods.

High-level LoRA and hierarchical fusion for enhanced micro-expression recognition

High-level LoRA and hierarchical fusion for enhanced micro-expression recognition

Robust biomimetic strain sensor based on butterfly wing-derived skeleton structure

A biomimetic strain sensor was designed and constructed based on Ir nanoparticles-modified multi-wall carbon nanotubes (Ir NPs@MWCNTs) and parallel Pt layer/dragon skin with carbonized butterfly wing patterns. This sensor exhibits high gauge factor (∼515.4), extensive tensile range (0%–96%), and swift response (∼300 ms), especially remarkable stability up to 60 000 cycles. The work mechanism has been proposed based on the experimental test and finite-element method. Some important applications such as human motion and micro-expression recognition have been confirmed using 3 × 3 flexible biomimetic sensor array.

Microexpression Recognition Method Based on ADP-DSTN Feature Fusion and Convolutional Block Attention Module

Microexpressions are subtle facial movements that occur within an extremely brief time frame, often revealing suppressed emotions. These expressions hold significant importance across various fields, including security monitoring and human–computer interaction. However, the accuracy of microexpression recognition is severely constrained by the inherent characteristics of these expressions. To address the issue of low detection accuracy regarding the subtle features present in microexpressions’ facial action units, this paper proposes a microexpression action unit detection algorithm, Attention-embedded Dual Path and Shallow Three-stream Networks (ADP-DSTN), that incorporates an attention-embedded dual path and a shallow three-stream network. First, an attention mechanism was embedded after each Bottleneck layer in the foundational Dual Path Networks to extract static features representing subtle texture variations that have significant weights in the action units. Subsequently, a shallow three-stream 3D convolutional neural network was employed to extract optical flow features that were particularly sensitive to temporal and discriminative characteristics specific to microexpression action units. Finally, the acquired static facial feature vectors and optical flow feature vectors were concatenated to form a fused feature vector that encompassed more effective information for recognition. Each facial action unit was then trained individually to address the issue of weak correlations among the facial action units, thereby facilitating the classification of microexpression emotions. The experimental results demonstrated that the proposed method achieved great performance across several microexpression datasets. The unweighted average recall (UAR) values were 80.71%, 89.55%, 44.64%, 80.59%, and 88.32% for the SAMM, CASME II, CAS(ME)3, SMIC, and MEGC2019 datasets, respectively. The unweighted F1 scores (UF1) were 79.32%, 88.30%, 43.03%, 81.12%, and 88.95%, respectively. Furthermore, when compared to the benchmark model, our proposed model achieved better performance with lower computational complexity, characterized by a Floating Point Operations (FLOPs) value of 1087.350 M and a total of 6.356 × 106 model parameters.

Counterfactual discriminative micro-expression recognition

Micro-expressions are spontaneous, rapid and subtle facial movements that can hardly be suppressed or fabricated. Micro-expression recognition (MER) is one of the most challenging topics in affective computing. It aims to recognize subtle facial movements which are quite difficult for humans to perceive in a fleeting period. Recently, many deep learning-based MER methods have been developed. However, how to effectively capture subtle temporal variations for robust MER still perplexes us. We propose a counterfactual discriminative micro-expression recognition (CoDER) method to effectively learn the slight temporal variations for video-based MER. To explicitly capture the causality from temporal dynamics hidden in the micro-expression (ME) sequence, we propose ME counterfactual reasoning by comparing the effects of the facts w.r.t. original ME sequences and the counterfactuals w.r.t. counterfactually-revised ME sequences, and then perform causality-aware prediction to encourage the model to learn those latent ME temporal cues. Extensive experiments on four widely-used ME databases demonstrate the effectiveness of CoDER, which results in comparable and superior MER performance compared with that of the state-of-the-art methods. The visualization results show that CoDER successfully perceives the meaningful temporal variations in sequential faces.

Transfer-learning enabled micro-expression recognition using dense connections and mixed attention

Micro-expression recognition (MER) is a challenging computer vision problem, where the limited amount of available training data and insufficient intensity of the facial expressions are among the main issues adversely affecting the performance of existing recognition models. To address these challenges, this paper explores a transfer–learning enabled MER model using a densely connected feature extraction module with mixed attention. Unlike previous works that utilize transfer learning to facilitate MER and extract local facial-expression information, our model relies on pretraining with three diverse macro-expression datasets and, as a result, can: (i) overcome the problem of insufficient sample size and limited training data availability, (ii) leverage (related) domain-specific information from multiple datasets with diverse characteristics, and (iii) improve the model adaptability to complex scenes. Furthermore, to enhance the intensity of the micro-expressions and improve the discriminability of the extracted features, the Euler video magnification (EVM) method is adopted in the preprocessing stage and then used jointly with a densely connected feature extraction module and a mixed attention mechanism to derive expressive feature representations for the classification procedure. The proposed feature extraction mechanism not only guarantees the integrity of the extracted features but also efficiently captures local texture cues by aggregating the most salient information from the generated feature maps, which is key for the MER task. The experimental results on multiple datasets demonstrate the robustness and effectiveness of our model compared to the state-of-the-art.

Multi-Modal Emotional Understanding in AI Virtual Characters: Integrating Micro-Expression-Driven Feedback within Context-Aware Facial Micro-Expression Processing Systems

To engage users, AI Virtual Characters must comprehend emotions. The paper develops and evaluates Chinese-specific context-aware facial micro-expression processing algorithms and feedback mechanisms to improve AI virtual characters' multi-modal emotional comprehension in Chinese culture. Specialized algorithms were used to collect and evaluate Chinese micro-expressions and assess AI virtual characters' emotional comprehension in user interactions. Chinese participants of various ages, genders, and places were recruited for micro-expression recognition to ensure cultural inclusion. A comprehensive method collected quantitative and qualitative data. We integrated interview and AI virtual character feedback with quantitative indicators like emotion recognition accuracy, user engagement, and micro-expression intensity. In the study, demographics affect emotion recognition accuracy and age, gender, and location-specific virtual avatars increase emotional resonance. A study demonstrated that context influences micro-expression interpretation, particularly in distinguishing urban and rural surprise and grief. Textual micro-expression recommendations and real-time AI character expression modifications increased accuracy and user experience. Micro-expressions, visual and auditory cues, and physiological reactions are linked, requiring multimodal signal processing for emotional awareness. Interactive virtual help, gaming, and education may benefit from culturally appropriate AI characters. Deep learning, multimodal fusion, and explainable AI are used in AI emotional interaction theory and technique. Finally, using Chinese cultural intricacies, our study improves AI virtual character multi-modal emotional comprehension. Culturally sensitive AI personalities and emotional AI technologies are developed using context-aware face micro-expression analysis algorithms and feedback systems.

Effects of the Acoustic-Visual Indoor Environment on Relieving Mental Stress Based on Facial Electromyography and Micro-Expression Recognition

People working and studying indoors for a long time can easily experience mental fatigue and stress. Virtual natural elements introduced into indoor environments can stimulate the human visual and auditory senses, thus relieving psychological stress. In this study, stress induction was achieved through noise playback, and the recovery effects on psychological stress of three set indoor environments, visual, auditory, and audio-visual, were investigated through changes in subjects’ facial expressions, electromyographic (EMG) signals, and subjective questionnaires. The experiment found that after stress induction through noise, the participants’ stress levels changed significantly. At this time, the subject scored low on the questionnaire, with electromyography readings higher than usual, and micro-expression recognition indicated negative emotions. After the restoration effects under the three working conditions of visual, auditory, and audio-visual combination, the average EMG values during the recovery period decreased from the baseline period (10 min after the subject acclimated to the environment), respectively. The results indicate that all three restoration conditions have the effect of relieving psychological stress, with the stress recovery effects of auditory and audio-visual conditions being superior to visual conditions. This study is of great significance for creating comfortable indoor environments and minimizing psychological pressure on indoor office workers.

Attention-Based Spatiotemporal-Aware Network for Fine-Grained Visual Recognition

On public benchmarks, current macro facial expression recognition technologies have achieved significant success. However, in real-life scenarios, individuals may attempt to conceal their true emotions. Conventional expression recognition often overlooks subtle facial changes, necessitating more fine-grained micro-expression recognition techniques. Different with prevalent facial expressions, weak intensity and short duration are the two main obstacles for perceiving and interpreting a micro-expression correctly. Meanwhile, correlations between pixels of visual data in spatial and channel dimensions are ignored in most existing methods. In this paper, we propose a novel network structure, the Attention-based Spatiotemporal-aware network (ASTNet), for micro-expression recognition. In ASTNet, we combine ResNet and ConvLSTM as a holistic framework (ResNet-ConvLSTM) to extract the spatial and temporal features simultaneously. Moreover, we innovatively integrate two level attention mechanisms, channel-level attention and spatial-level attention, into the ResNet-ConvLSTM. Channel-level attention is used to discriminate the importance of different channels because the contributions for the overall presentation of micro-expression vary between channels. Spatial-level attention is leveraged to dynamically estimate weights for different regions due to the diversity of regions’ reflections to micro-expression. Extensive experiments conducted on two benchmark datasets demonstrate that ASTNet achieves performance improvements of 4.25–16.02% and 0.79–12.93% over several state-of-the-art methods.

Real-time microexpression recognition in educational scenarios using a dual-branch continuous attention network

Real-time microexpression recognition in educational scenarios using a dual-branch continuous attention network

Highly Sensitive, Degradable, and Rapid Self-Healing Hydrogel Sensor with Semi-Interpenetrating Network for Recognition of Micro-Expressions.

Flexible conductive hydrogels have revolutionized the lives and are widely applied in health monitoring and wearable electronics as a new generation of sensing materials. However, the inherent low mechanical strength, sensitivity, and lack of rapid self-healing capacity results in their short life, poor detection accuracy, and environmental pollution. Inspired by the molecular structure of bone and its chemical characteristics, a novel fully physically cross-linked conductive hydrogel is fabricated by the introduction of nanohydroxyapatite (HAp) as the dynamic junction points. In detail, the dynamically cross-linked network, including multiple physical interactions, provides it with rapid self-healing ability and excellent mechanical properties (elongation at break (>1200%), tensile strength (174kPa), and resilience (92.61%)). Besides, the ions (Cl-, Li+, Ca2+) that move freely within the system impart outstanding electrical conductivity (2.46±0.15 S m-1), high sensitivity (gauge factor, GF>8), good antifreeze (-40.2°C), and humidity properties. The assembled sensor can be employed to sensitively detect various large human motions and subtle changes in behavior (facial expressions, speech recognition). Meanwhile, the hydrogel sensor can also degrade in phosphate-buffered saline solution without causing any environmental pollution. Therefore, the designed hydrogels may become a promising candidate material in the future potential applications for smart wearable sensors and electronic skin.

SDGSA: a lightweight shallow dual-group symmetric attention network for micro-expression recognition

Recognizing micro-expressions (MEs) as subtle and transient forms of human emotional expressions is critical for accurately judging human feelings. However, recognizing MEs is challenging due to their transient and low-intensity characteristics. This study develops a lightweight shallow dual-group symmetric attention network (SDGSA) to address the limitations of existing methods in capturing the subtle features of MEs. This network takes the optical flow features as inputs, extracting ME features through a shallow network and performing finer feature segmentation in the channel dimension through a dual-group strategy. The goal is to focus on different types of facial information without disrupting facial symmetry. Moreover, this study implements a spatial symmetry attention module, focusing on extracting facial symmetry features to emphasize further the symmetric information of the left and right sides of the face. Additionally, we introduce the channel blending technique to optimize the information fusion between different channel features. Extensive experiments on SMIC, CASME II, SAMM, and 3DB-combined mainstream ME datasets demonstrate that the proposed SDGSA method outperforms the metrics of current state-of-the-art methods. As shown by ablation experimental results, the proposed dual-group symmetric attention module outperforms classical attention modules, such as the convolutional block attention module, squeeze-and-excitation, efficient channel attention, spatial group-wise enhancement, and multi-head self-attention. Importantly, SDGSA maintained excellent performance while having only 0.278 million parameters. The code and model are publicly available at https://github.com/YZY980123/SDGSA.

A review of research on micro-expression recognition algorithms based on deep learning

A review of research on micro-expression recognition algorithms based on deep learning

Facial micro-expression recognition based on motion magnification network and graph attention mechanism

Micro-expression is extensively studied due to their ability to fully reflect individuals' genuine emotions. However, accurate micro-expression recognition is a challenging task due to the subtle motion of facial muscle. Therefore, this paper introduces a Graph Attention Mechanism-based Motion Magnification Guided Micro-Expression Recognition Network (GAM-MM-MER) to amplify delicate muscle motions and focus on key facial landmarks. First, we propose a Swin Transformer-based network for micro-expression motion magnification (ST-MEMM) to enhance the subtle motions in micro-expression videos, thereby unveiling imperceptible facial muscle movements. Then, we propose a graph attention mechanism-based network for micro-expression recognition (GAM-MER), which optimizes facial key area maps and prioritizes adjacent nodes crucial for mitigating the influence of noisy neighbors, while attending to key feature information. Finally, experimental evaluations conducted on the CASME II and SAMM datasets demonstrate the high accuracy and effectiveness of the proposed network compared to state-of-the-art approaches. The results of our network exhibit significant superiority over existing methods. Furthermore, ablation studies provide compelling evidence of the robustness of our proposed network, substantiating its efficacy in micro-expression recognition.

HTNet for micro-expression recognition

Facial expression is related to facial muscle contractions and different muscle movements correspond to different emotional states. For micro-expression recognition, the muscle movements are usually subtle, which has a negative impact on the performance of current facial emotion recognition algorithms. Most existing methods use self-attention mechanisms to capture relationships between tokens in a sequence, but they do not take into account the inherent spatial relationships between facial landmarks. This can result in sub-optimal performance on micro-expression recognition tasks. Therefore, learning to recognize facial muscle movements is a key challenge in the area of micro-expression recognition. In this paper, we propose a Hierarchical Transformer Network (HTNet) to identify critical areas of facial muscle movement. HTNet includes two major components: a transformer layer that leverages the local temporal features and an aggregation layer that extracts local and global semantical facial features. Specifically, HTNet divides the face into four different facial areas: left lip area, left eye area, right eye area and right lip area. The transformer layer is used to focus on representing local minor muscle movement with local self-attention in each area. The aggregation layer is used to learn the interactions between eye areas and lip areas. The experiments on four publicly available micro-expression datasets show that the proposed approach outperforms previous methods by a large margin. The codes and models are available at: https://github.com/wangzhifengharrison/HTNet.

Exploring holistic discriminative representation for micro-expression recognition via contrastive learning

Recently, deep learning-based micro-expression recognition (MER) has been remarkably successful in the affective computing and computer vision communities. However, the most challenging issue that hinders the performance of MER is low intensity. Instead of forcefully transforming the input from micro-expressions to exaggerated micro-expressions by a fixed video motion magnification factor, our approach introduces a sophisticated pretext task with an intensity-agnostic strategy to enhance the discriminative capacity of each micro-expression sample holistically through contrastive transfer learning. This strategy enables us to progressively transfer knowledge and leverage the rich facial expression information from macro-expression samples. In addition, we reconsider that the core of the MER task is to refine and incorporate the instance-level and class-level discriminative features from the initial indistinguishable information. As a result, we jointly merge the two views to learn a holistic-level representation. Simultaneously, to ensure a strong association and guidance between the instance-level view and the class-level view, we maintain their consistency through an alignment loss. The results showed that the proposed method could significantly improve the performance of MER on CASME II, SAMM, SMIC, and CAS(ME)3 datasets.

Micro-expression recognition based on euler video magnification and 3D residual network under imbalanced sample

A student’s verbal behavior plays a crucial role in education, while nonverbal behavior, such as micro-expressions, significantly improves teaching quality. To address the problem of small facial expression movements, imbalanced data categories, and lack of temporal information in static expressions, a micro-expression recognition method is proposed based on Eulerian Video Magnification (EVM) and a 3D Residual Network (3D ResNet) under imbalanced samples. Firstly, face detection in the Dlib library is used to locate the face in the micro-expression video sample and crop it. Secondly, the EVM is used to magnify the motion features in micro-expressions. Then, the 3D ResNet is used to extract spatio-temporal information from micro-expression video samples, and the Cyclical Focal Loss (CFL) function is introduced in the network training process to solve the class imbalance problem in micro-expression datasets. Finally, the roles of the EVM and the CFL function in recognizing micro-expressions by the 3D ResNet are analyzed. The experimental results on the Spontaneous Micro-expression Database (SMIC) and Chinese Academy of Sciences Micro-expression Database II (CASME II) demonstrate the effectiveness and superiority of this method. The proposed method can assist in teaching evaluation and promote the development of smart classrooms, and further research is needed on the storage and computing of the proposed method on devices.

Micro-expression recognition based on a novel GCN-transformer cooperation model for IoT-eHealth

If a person is truly healthy, his/her well-being encompasses both physical and psychological health. However, the existing IoT-eHealth system typically focus only on monitoring the user’s physical data through various sensors, neglecting their mental state. To enhance the intelligence level of IoT-eHealth system and enable it to have the psychological monitoring ability, a novel collaborative model based on Graph Convolutional Network (GCN) and Transformer is designed for Micro-Expression (ME) recognition in this paper. Firstly, facial information within each frame is transformed into a Spatial Topological Relationship Graph (STRG) by using facial landmarks detection and psychological relationship of local patches. Then, in order to automatically aggregate the key information on facial patches that contribute to ME recognition from the structured graph data, a Hierarchical Adaptive Graph Pooling (HAGP) module is designed for obtaining discriminative frame-level feature based on GCN utilizing graph structure and vertex global dependencies. Finally, in order to model the long-term dependencies among frames and capture the key frame-level features that are beneficial for ME recognition, a Temporal Sensitive Self-Attention (TSSA) mechanism is designed, and a novel Temporal Sensitive Transformer (TST) encoder is constructed based on TSSA to explore the evolution law of facial patterns and obtain discriminative video-level features that are helpful for ME recognition. In the comparative experiments of standard dataset verification and practical dataset testing, designed collaborative model is superior to other methods and can achieve the highest recognition accuracy, which almost can meet the application requirements of IoT-eHealth system.

Micro-expression recognition based on contextual transformer networks

Micro-expression recognition based on contextual transformer networks

Facial micro-expression recognition using stochastic graph convolutional network and dual transferred learning

Micro-expression recognition (MER) has drawn increasing attention due to its wide application in lie detection, criminal detection and psychological consultation. However, the best recognition accuracy on recent public dataset is still low compared to the accuracy of macro-expression recognition. In this paper, we propose a novel graph convolution network (GCN) for MER achieving state-of-the-art accuracy. Different to existing GCN with fixed graph structure, we define a stochastic graph structure in which some neighbors are selected randomly. As shown by numerical examples, randomness enables better feature characterization while reducing computational complexity. The whole network consists of two branches, one is the spatial branch taking micro-expression images as input, the other is the temporal branch taking optical flow images as input. Because the micro-expression dataset does not have enough images for training the GCN, we employ the transfer learning mechanism. That is, different stochastic GCNs (SGCN) have been trained by the macro-expression dataset in the source network. Then the well-trained SGCNs are transferred to the target network. It is shown that our proposed method achieves the state-of-art performance on all four well-known datasets. This paper explores stochastic GCN and transfer learning with this random structure in the MER task, which is of great importance to improve the recognition performance.