IMPROVING MICRO-EXPRESSION RECOGNITION WITH AN ENHANCED DESCRIPTOR COMBINING GW LBP, TGMH, AND WT

Micro-expressions (MEs) are rapid, involuntary facial expressions which reveal emotions that people do not intend to show. Studying MEs is valuable as recognizing them has many important applications, particularly in forensic science and psychotherapy. However, analyzing spontaneous MEs is very challenging due to their short duration and low intensity. Automatic ME analysis includes two tasks: ME spotting and ME recognition. For ME spotting, previous studies have focused on posed rather than spontaneous videos. For ME recognition, the performance of previous studies is low. To address these challenges, we make the following contributions: (i) We propose the first method for spotting spontaneous MEs in long videos (by exploiting feature difference contrast). This method is training free and works on arbitrary unseen videos. (ii) We present an advanced ME recognition framework, which outperforms previous work by a large margin on two challenging spontaneous ME databases (SMIC and CASMEII). (iii) We propose the first automatic ME analysis system (MESR), which can spot and recognize MEs from spontaneous video data. Finally, we show our method outperforms humans in the ME recognition task by a large margin, and achieves comparable performance to humans at the very challenging task of spotting and then recognizing spontaneous MEs.

The micro-expression (ME) processing characteristics of patients with depression has been studied but has not been investigated in people with subthreshold depression. Based on this, by adopting the ecological MEs recognition paradigm, this study aimed to explore ME recognition in people with subthreshold depression. A 4 (background expression: happy, neutral, sad and fearful) × 4 (ME: happy, neutral, sad, and fearful) study was designed; two groups of participants (experimental group with subthreshold depression vs. healthy control group, 32 participants in each group) were asked to complete the ecological ME recognition task, and the corresponding accuracy (ACC) and reaction time (RT) were analyzed. Results: (1) Under different background conditions, recognizing happy MEs had the highest ACC and shortest RT. (2) There was no significant difference in the ACC and RT between experimental and control groups. (3)In different contexts, individuals with subthreshold depression tended to misjudge neutral, sad, and fearful MEs as happy, while neutral MEs were misjudged as sad and fearful. (4) The performance of individuals with subthreshold depression in the ecological ME recognition task were influenced by the type of ME; they showed highest ACC and shortest RT when recognizing happy MEs (vs. the other MEs). Conclusions: (1) The performance of individuals’ ecological ME recognition were influenced by the background expression, and this embodied the need for ecological ME recognition. (2) Individuals with subthreshold depression showed normal ecological ME recognition ability. (3) In terms of misjudgment, individuals with subthreshold depression showed both positive and negative bias, when completing the ecological ME recognition task. (4) Compared with the other MEs, happy MEs showed an advantage recognition effect for individuals with subthreshold depression who completed the ecological ME recognition task.

Micro-expression (ME) recognition has attracted numerous interests within the computer vision circle in different contexts particularly, localization, magnification, and recognition. Challenges in these areas remain relevant due to the nature of ME’s split-second transition with minute intensity levels. In this paper, a comprehensive state-of-the-art analysis of ME recognition and detection challenges are provided. Contemporary solutions are categorized into low-level, mid-level, and high-level solutions with a review of their characteristics and performances. This paper also provides possible extensions to basic methods, highlight, and predict emerging trends. A thorough analysis of mainstream ME datasets is also provided by elucidating each of their advantages and limitations. This survey gives readers an understanding of ME recognition and an appreciation of future research direction in ME recognition systems.

Due to the short duration and low intensity, the efficient feature learning is a big challenge for robust facial microexpression (ME) recognition. To achieve the diverse and spatial relation representation, this paper proposes a simple yet effective micro-expression recognition method based on multiscale convolutional fusion and capsule network (MCFCN). Firstly, the apex frame in a ME clip is located by computing the pixel difference of frames, and then the apex frame is processed by an optical flow operator. Secondly, a multi-scale fusion module is introduced to capture diverse ME related details. Then, the micro-expression features are fed into the capsule network for a good description about spatial relation. Finally, the entire ME recognition model is trained and verified on three popular benchmarks (SAMM, SMIC and CASMEII) using the associated standard evaluation protocols. Experimental results show that our method based on MCFCN is superior to the works based on pervious capsule network or other state-of-the-art CNN models.

Micro-expression recognition based on a novel GCN-transformer cooperation model for IoT-eHealth

With the rapid growth of heritage tourism and increasing demands for museums to attract diverse audiences, understanding visitors’ emotional engagement and satisfaction has gained significant importance. Capturing and analyzing fleeting emotions and feelings in real-time provides critical insights into visitor behavior, offering opportunities to enhance visitor experiences, improve satisfaction, and support heritage conservation efforts. This underscores the need for innovative, technology-driven solutions to effectively evaluate and understand these emotional responses. Micro-expressions (MEs), fleeting and subtle FEs that reflect underlying emotions, are of significant importance for understanding human behavior. Automatic recognition of MEs is critical for applications that aim to discern true emotions, such as visitor experience analysis in museum settings. However, existing ME datasets are often limited to laboratory settings, which do not reflect the complexity and variability of real-world situations. To address this issue, we developed a novel spontaneous ME dataset in a museum setting (SMEM) comprising 127 participants and 344 ME video clips that were categorized into four basic types (negative, positive, surprise, and others). The onset frame, apex frame, offset frame, and action units (AUs) of each ME sample were annotated to accurately describe the facial movements. For baseline evaluation, we adopted optical flow to extract hand-crafted features of facial movements for ME apex spotting and designed a shallow CNN model with a cross-attention module to perform ME recognition with the optical flow as input. Our results show that the CNN method achieved a promising result for ME recognition in a real-world environment. The study provides a valuable contribution to the field of automatic ME recognition in real-world settings and highlights the potential of using ME recognition in museum visitor experience analysis.

Micro-expression (ME) is a subtle and involuntary facial expression that reveals the human's concealed emotion. Psychology studies pointed out that research of ME can build potential applications in many fields. Therefore, ME recognition (MER), one of the two main ME analysis tasks, has been becoming an attractive topic recently. However, the work of MER is still needed to consider due to several limitations related to performance and dataset. This paper proposes a feature fusion between optical flow and dynamic image to create a robust ME representation, which can be learned effectively from deep learning techniques. Experiments from two public datasets, CASME-II and SAMM, show that our method obtains higher performance than several existing studies and is very promising for future research. CCS CONCEPTS •Computing methodologies∼Object recognition

Micro-Expressions (MEs) are involuntary and im-perceptible facial movements that reflect the underlying emotions and inner activities. Recently, ME recognition technology has been widely used in several fields such as medical treatment. Due to the subtle variations among the video sequence and the limited training data, the ME recognition task still remains a challenging problem. Existing methods tend to address the ME recognition problem from two aspects: (1) Data augmentation and (2) Expression signal amplification. Few works realize the importance of temporal variation hidden in the ME sequence. Based on the above observation, we propose a Graph Contrastive Learning (GCL) framework to effectively perceive subtle temporal variation for robust ME recognition. Specifically, the strong spatial feature representation is captured through the transformer-based ME feature encoder. Then, the proposed GCL builds the graph structure for the ME sequence and introduces the graph convolution to model the temporal relationship. To capture and highlight the temporal variation hidden in the ME sequence, a contrastive learning framework is designed to discriminately learn the differences between the normal and the abnormal ME samples. Both quantitative and qualitative experimental results show the effectiveness and superiority of our method compared with the prior state-of-the-arts.

Facial Micro-Expressions (MEs) are spontaneous, involuntary facial movements when a person experiences an emotion but deliberately or unconsciously attempts to conceal his or her genuine emotions. Recently, ME recognition has attracted increasing attention due to its potential applications such as clinical diagnosis, business negotiation, interrogations, and security. However, it is expensive to build large scale ME datasets, mainly due to the difficulty of inducing spontaneous MEs. This limits the application of deep learning techniques which require lots of training data. In this paper, we propose a simple, efficient yet robust descriptor called Extended Local Binary Patterns on Three Orthogonal Planes (ELBPTOP) for ME recognition. ELBPTOP consists of three complementary binary descriptors: LBPTOP and two novel ones Radial Difference LBPTOP (RDLBPTOP) and Angular Difference LBPTOP (ADLBPTOP), which explore the local second order information along the radial and angular directions contained in ME video sequences. ELBPTOP is a novel ME descriptor inspired by unique and subtle facial movements. It is computationally efficient and only marginally increases the cost of computing LBPTOP, yet is extremely effective for ME recognition. In addition, by firstly introducing Whitened Principal Component Analysis (WPCA) to ME recognition, we can further obtain more compact and discriminative feature representations, then achieve significantly computational savings. Extensive experimental evaluation on three popular spontaneous ME datasets SMIC, CASME II and SAMM show that our proposed ELBPTOP approach significantly outperforms the previous state-of-the-art on all three single evaluated datasets and achieves promising results on cross-database recognition. Our code will be made available.

Facial micro-expression (ME) can disclose genuine and concealed human feelings. It makes MEs extensively useful in real-world applications pertaining to affective computing and psychology. Unfortunately, they are induced by subtle facial movements for a short duration of time, which makes the ME recognition, a highly challenging problem even for human beings. In automatic ME recognition, the well-known features encode either incomplete or redundant information, and there is a lack of sufficient training data. The proposed method, Micro-Expression Recognition by Analysing Spatial and Temporal Characteristics, MERASTC mitigates these issues for improving the ME recognition. It compactly encodes the subtle deformations using action units (AUs), landmarks, gaze, and appearance features of all the video frames while preserving most of the relevant ME information. Furthermore, it improves the efficacy by introducing a novel neutral face normalization for ME and initiating the utilization of gaze features in deep learning based ME recognition. The features are provided to the 2D convolutional neural network that jointly analyses the spatial and temporal behavior for correct ME classification. Experimental results on the publicly available datasets indicate that the proposed method exhibits better performance than the well-known methods.

A micro-expression (ME) is a kind of involuntary facial expressions, which commonly occurs with subtle intensity. The accurately recognition ME, a. k. a. micro-expression recognition (MER), has a number of potential applications, e.g., interrogation and clinical diagnosis. Therefore, the subject has received a high level of attention among researchers in affective computing and pattern recognition communities. In this paper, we proposed a straightforward and effective deep learning method called uncertainty-aware magnification-robust networks (UAMRN) for MER, which attempts to address two key issues in MER including the low intensity of ME and imbalance of ME samples. Specifically, to better distinguish subtle ME movements, we reconstructed a new sequence by magnifying the ME intensity. Furthermore, a sparse self-attention (SSA) block was implemented which rectifies the standard self-attention with locality sensitive hashing (LSH), resulting in the suppression of artefacts generated during magnification. On the other hand, for the class imbalance problem, we guided the network optimization based on the confidence about the estimation, through which the samples from rare classes were allotted greater uncertainty and thus trained more carefully. We conducted the experiments on three public ME databases, i.e., CASME II, SAMM and SMIC-HS, the results of which demonstrate improvement compared to recent state-of-the-art MER methods.

Micro-Expression (ME) recognition is a hot topic in computer vision as it presents a gateway to capture and understand daily human emotions. It is nonetheless a challenging problem due to ME typically being transient (lasting less than 200 ms) and subtle. Recent advances in machine learning enable new and effective methods to be adopted for solving diverse computer vision tasks. In particular, the use of deep learning techniques on large datasets outperforms classical approaches based on classical machine learning which rely on hand-crafted features. Even though available datasets for spontaneous ME are scarce and much smaller, using off-the-shelf Convolutional Neural Networks (CNNs) still demonstrates satisfactory classification results. However, these networks are intense in terms of memory consumption and computational resources. This poses great challenges when deploying CNN-based solutions in many applications, such as driver monitoring and comprehension recognition in virtual classrooms, which demand fast and accurate recognition. As these networks were initially designed for tasks of different domains, they are over-parameterized and need to be optimized for ME recognition. In this paper, we propose a new network based on the well-known ResNet18 which we optimized for ME classification in two ways. Firstly, we reduced the depth of the network by removing residual layers. Secondly, we introduced a more compact representation of optical flow used as input to the network. We present extensive experiments and demonstrate that the proposed network obtains accuracies comparable to the state-of-the-art methods while significantly reducing the necessary memory space. Our best classification accuracy was 60.17% on the challenging composite dataset containing five objectives classes. Our method takes only 24.6 ms for classifying a ME video clip (less than the occurrence time of the shortest ME which lasts 40 ms). Our CNN design is suitable for real-time embedded applications with limited memory and computing resources.

Micro-expression (ME) is an extremely quick and uncontrollable facial movement that lasts for 40–200 ms and reveals thoughts and feelings that an individual attempts to cover up. Though much more difficult to detect and recognize, ME recognition is similar to macro-expression recognition in that it is influenced by facial features. Previous studies suggested that facial attractiveness could influence facial expression recognition processing. However, it remains unclear whether facial attractiveness could also influence ME recognition. Addressing this issue, this study tested 38 participants with two ME recognition tasks in a static condition or dynamically. Three different MEs (positive, neutral, and negative) at two attractiveness levels (attractive, unattractive). The results showed that participants recognized MEs on attractive faces much quicker than on unattractive ones, and there was a significant interaction between ME and facial attractiveness. Furthermore, attractive happy faces were recognized faster in both the static and the dynamic conditions, highlighting the happiness superiority effect. Therefore, our results provided the first evidence that facial attractiveness could influence ME recognition in a static condition or dynamically.

ME-PLAN: A deep prototypical learning with local attention network for dynamic micro-expression recognition

A weighted feature extraction method based on temporal accumulation of optical flow for micro-expression recognition