Micro-expression Analysis Research Articles

Multi-Modal Emotional Understanding in AI Virtual Characters: Integrating Micro-Expression-Driven Feedback within Context-Aware Facial Micro-Expression Processing Systems

To engage users, AI Virtual Characters must comprehend emotions. The paper develops and evaluates Chinese-specific context-aware facial micro-expression processing algorithms and feedback mechanisms to improve AI virtual characters' multi-modal emotional comprehension in Chinese culture. Specialized algorithms were used to collect and evaluate Chinese micro-expressions and assess AI virtual characters' emotional comprehension in user interactions. Chinese participants of various ages, genders, and places were recruited for micro-expression recognition to ensure cultural inclusion. A comprehensive method collected quantitative and qualitative data. We integrated interview and AI virtual character feedback with quantitative indicators like emotion recognition accuracy, user engagement, and micro-expression intensity. In the study, demographics affect emotion recognition accuracy and age, gender, and location-specific virtual avatars increase emotional resonance. A study demonstrated that context influences micro-expression interpretation, particularly in distinguishing urban and rural surprise and grief. Textual micro-expression recommendations and real-time AI character expression modifications increased accuracy and user experience. Micro-expressions, visual and auditory cues, and physiological reactions are linked, requiring multimodal signal processing for emotional awareness. Interactive virtual help, gaming, and education may benefit from culturally appropriate AI characters. Deep learning, multimodal fusion, and explainable AI are used in AI emotional interaction theory and technique. Finally, using Chinese cultural intricacies, our study improves AI virtual character multi-modal emotional comprehension. Culturally sensitive AI personalities and emotional AI technologies are developed using context-aware face micro-expression analysis algorithms and feedback systems.

A software package for the investigation of spatio-temporal feature descriptors

At present time researchers investigate approaches to the automatic facial micro-expression analysis to implement the possibility of the recognition of true human emotions from the image sequence of his face. The modern technologies are based on the use of complex mathematical algorithms with many adjusted parameters; in particular, such algorithms are the algorithms of spatio-temporal feature descriptors. The scientists who are not technical specialists, e.g., psychologists or psychotherapists, when investigating these algorithms may encounter difficulties in writing the program code. Therefore, it became necessary to develop a special software package for the investigation of spatio-temporal feature descriptors. Aim of the work is development of a special software package for studying various spatial and temporal descriptors of facial micromovement features in video images A software package for the investigation of spatio-temporal feature descriptors with a graphical interface is developed. The software package makes it possible to investigate the influence of the selection of the descriptor (LBP-TOP, FHOOF or FHOFO) and its parameters on the micro-facial movement detection accuracy for different input datasets of human face image sequences. The developed software package for the investigation of spatio-temporal feature descriptors can be applied as a software tool for carrying out investigations in the facial micro-expression analysis field.

Deep learning-based Facial Micro-Expression Analysis: A Survey of Datasets and Algorithms

Deep learning-based Facial Micro-Expression Analysis: A Survey of Datasets and Algorithms

Protocol for computer vision based micro-expression analysis to identify stroke in limited resource settings

Protocol for computer vision based micro-expression analysis to identify stroke in limited resource settings

SFAMNet: A scene flow attention-based micro-expression network

Tremendous progress has been made in facial Micro-Expression (ME) spotting and recognition; however, most works have focused on either spotting or recognition tasks on the 2D videos. Until recently, the estimation of the 3D motion field (a.k.a scene flow) for the ME has only become possible after the release of the multi-modal ME dataset. In this paper, we propose the first Scene Flow Attention-based Micro-expression Network, namely SFAMNet. It takes the scene flow computed using the RGB-D flow algorithm as the input and predicts the spotting confidence score and emotion labels. Specifically, SFAMNet is an attention-based end-to-end multi-stream multi-task network devised to spot and recognize the ME. Besides that, we present a data augmentation strategy to alleviate the small sample size problem during network learning. Extensive experiments are performed on three tasks: (i) ME spotting; (ii) ME recognition; and (iii) ME analysis on the multi-modal CAS(ME)3 dataset. Empirical results indicate that depth is vital in capturing the ME information and the effectiveness of the proposed approach. Our source code is publicly available at https://github.com/genbing99/SFAMNet.

An Overview of Facial Micro-Expression Analysis: Data, Methodology and Challenge

Facial micro-expressions indicate brief and subtle facial movements that appear during emotional communication. In comparison to macro-expressions, micro-expressions are more challenging to be analyzed due to the short span of time and the fine-grained changes. In recent years, micro-expression recognition (MER) has drawn much attention because it can benefit a wide range of applications, e.g. police interrogation, clinical diagnosis, depression analysis, and business negotiation. In this survey, we offer a fresh overview to discuss new research directions and challenges these days for MER tasks. For example, we review MER approaches from three novel aspects: macro-to-micro adaptation, recognition based on key apex frames, and recognition based on facial action units. Moreover, to mitigate the problem of limited and biased ME data, synthetic data generation is surveyed for the diversity enrichment of micro-expression data. Since micro-expression spotting can boost micro-expression analysis, the state-of-the-art spotting works are also introduced in this paper. At last, we discuss the challenges in MER research and provide potential solutions as well as possible directions for further investigation.

Micro-expression spotting with multi-scale local transformer in long videos

Micro-expression analysis by computer vision techniques has attracted much attention as it can reveal the human emotions automatically. Among the analysis tasks, the temporal spotting is the most challenging task for achieving expression-aware frames from long video sequences. Compared to the well studied recognition task, more researches need to be devoted to the spotting task for further improving the performance and benefiting the subsequent tasks. So, in this paper, we propose a convolutional transformer based deep model for micro-expression spotting in long video sequences. A 3D convolutional subnetwork is firstly employed to extract the visual features from the temporal frames in a fixed-size sliding window of original video sequence. Then a multi-scale local transformer module is designed based on the visual features to model the correlation between frames in a local window. By leveraging the correlation information, the description of face movement becomes more representative for various-duration micro-expressions. Finally, the multi-head classifier and the corresponding estimator are jointly combined to predict the temporal position for spotting micro-expressions. The proposed method is evaluated on two publicly-available datasets, namely CAS(ME)2 and SAMM-LV, and achieves the promising performance of 0.2770 F1-score on SAMM-LV and 0.1373 F1-score on CAS(ME)2. The code is publicly available on GitHub (https://github.com/xiazhaoqiang/MULT-MicroExpressionSpot).

Facial Micro Expression Recognition for Feature Point Tracking using Apex Frames on CASME II Database

Micro-expressions are facial expressions that occur inadvertently to hide true feelings (emotional leaks). Although previous studies used the entire face area and all frames in the video dataset, this resulted in relatively long computation time and data redundancy. The main contribution of this research is to apply recognition micro-expression analysis using a comparison of apex frames with manual (handcrafted) and random sampling of frames and applying feature point tracking to the brow area and corners of the lips. The method for forming feature points in the facial area uses Discriminative Response Map Fitting (DRMF), then facial feature points are tracked using Kanade-Lucas-Tomasi (KLT). This feature point tracking produces motion feature data as feature extraction data. Finally, a comparative analysis of the classification method using the Support Vector Machine (SVM) and MLP-Backpropagation was conducted using the CASME II dataset. The experimental results of this study show significant results with an accuracy of 81.3% on MLP-Backpropagation and an average computing time of 1.45 seconds for each video. From the results of this study, information on the apex phase can contribute information that is very important for facial micro-expression recognition.

Deep 3D Convolutional Neural Network for Facial Micro-Expression Analysis from Video Images

Micro-expression is the involuntary emotion of the human that reflects the genuine feelings that cannot be hidden. Micro-expression is exhibited by facial expressions that last for a short duration and have very low intensity. Because of these reasons, micro-expression recognition is a challenging task. Recent research on the application of 3D convolutional neural networks (CNNs) has gained much popularity for video-based micro-expression analysis. For this purpose, both spatial as well as temporal features are of great importance to achieve high accuracies. The real possibly suppressed emotions of a person are valuable information for a variety of applications, such as in security, psychology, neuroscience, medicine and many other disciplines. This paper proposes a 3D CNN model architecture which is able to extract spatial and temporal features simultaneously. Thereby, the selection of the frame sequence plays a crucial role, since the emotions are only distinctive in a subset of the frames. Thus, we employ a novel pre-processing technique to select the Apex frame sequence from the entire video, where the timestamp of the most pronounced emotion is centered within this sequence. After an extensive evaluation including many experiments, the results show that the train–test split evaluation is biased toward a particular split and cannot be recommended in case of small and imbalanced datasets. Instead, a stratified K-fold evaluation technique is utilized to evaluate the model, which proves to be much more appropriate when using the three benchmark datasets CASME II, SMIC, and SAMM. Moreover, intra-dataset as well as cross-dataset evaluations were conducted in a total of eight different scenarios. For comparison purposes, two networks from the state of the art were reimplemented and compared with the presented architecture. In stratified K-fold evaluation, our proposed model outperforms both reimplemented state-of-the-art methods in seven out of eight evaluation scenarios.

Spot-then-Recognize: A Micro-Expression Analysis Network for Seamless Evaluation of Long Videos

Facial Micro-Expressions (MEs) reveal a person’s hidden emotions in high stake situations within a fraction of a second and at a low intensity. The broad range of potential real-world applications that can be applied has drawn considerable attention from researchers in recent years. However, both spotting and recognition tasks are often treated separately. In this paper, we present Micro-Expression Analysis Network (MEAN), a shallow multi-stream multi-output network architecture comprising of task-specific (spotting and recognition) networks that is designed to effectively learn a meaningful representation from both ME class labels and location-wise pseudo-labels. Notably, this is the first known work that addresses ME analysis on long videos using a deep learning approach, whereby ME spotting and recognition are performed sequentially in a two-step procedure: first spotting the ME intervals using the spotting network, and proceeding to predict their emotion classes using the recognition network. We report extensive benchmark results on the ME analysis task on both short video datasets (CASME II, SMIC-E-HS, SMIC-E-VIS, and SMIC-E-NIR), and long video datasets (CAS(ME)2 and SAMMLV); the latter in particular demonstrates the capability of the proposed approach under unconstrained settings. Besides the standard measures, we promote the usage of fairer metrics in evaluating the performance of a complete ME analysis system. We also provide visual explanations of where the network is “looking” and showcasing the effectiveness of inductive transfer applied during network training. An analysis is performed on the in-the-wild dataset (MEVIEW) to open up further research into real-world scenarios.

A Review of Automated Micro-expression Analysis

Micro-expression is a type of facial expression that is manifested for a very short duration. It is difficult to recognize the expression manually because it involves very subtle facial movements. Such expressions often occur unconsciously, and therefore are defined as a basis to help identify the real human emotions. Hence, an automated approach to micro-expression recognition has become a popular research topic of interest recently. Historically, the early researches on automated micro-expression have utilized traditional machine learning methods, while the more recent development has focused on the deep learning approach. Compared to traditional machine learning, which relies on manual feature processing and requires the use of formulated rules, deep learning networks produce more accurate micro-expression recognition performances through an end-to-end methodology, whereby the features of interest were extracted optimally through the training process, utilizing a large set of data. This paper reviews the developments and trends in micro-expression recognition from the earlier studies (hand-crafted approach) to the present studies (deep learning approach). Some of the important topics that will be covered include the detection of micro-expression from short videos, apex frame spotting, micro-expression recognition as well as performance discussion on the reviewed methods. Furthermore, major limitations that hamper the development of automated micro-expression recognition systems are also analyzed, followed by recommendations of possible future research directions.

Pilot Experiments for Multi-Criteria Human Comfort-Driven Structural Glass Design Assessment

Civil engineering design and industry are continuously evolving with the support of advancements in technology. Digital tools are able to assist designers in solving several issues with more accuracy and minimized efforts. In parallel, maximization of human comfort is a target for various design procedures, where mathematical models and standardized protocols are conventionally used to optimize well-being of customers. Major challenges and troubles can indeed derive, structurally speaking, from human reactions, which are related to a multitude of aspects, and may further enforced by slender / transparent glass components. The so-called “emotional architecture” and its nervous feelings are intrinsic part of the issue, and hence the mutual interaction of objective and subjective parameters can make complex the building design optimization. This paper presents some recent studies in which human comfort for glass structures occupants is quantitatively measured, both with the support of remote digital technologies based on facial micro-expression analysis and in-field experiments able to capture kinematic and biometric parameters for customers moving in glass environments.

Optimal Compact Network for Micro-Expression Analysis System

Micro-expression analysis is the study of subtle and fleeting facial expressions that convey genuine human emotions. Since such expressions cannot be controlled, many believe that it is an excellent way to reveal a human’s inner thoughts. Analyzing micro-expressions manually is a very time-consuming and complicated task, hence many researchers have incorporated deep learning techniques to produce a more efficient analysis system. However, the insufficient amount of micro-expression data has limited the network’s ability to be fully optimized, as overfitting is likely to occur if a deeper network is utilized. In this paper, a complete deep learning-based micro-expression analysis system is introduced that covers the two main components of a general automated system: spotting and recognition, with also an additional element of synthetic data augmentation. For the spotting part, an optimized continuous labeling scheme is introduced to spot the apex frame in a video. Once the apex frames have been recognized, they are passed to the generative adversarial network to produce an additional set of augmented apex frames. Meanwhile, for the recognition part, a novel convolutional neural network, coined as Optimal Compact Network (OC-Net), is introduced for the purpose of emotion recognition. The proposed system achieved the best F1-score of 0.69 in categorizing the emotions with the highest accuracy of 79.14%. In addition, the generated synthetic data used in the training phase also contributed to performance improvement of at least 0.61% for all tested networks. Therefore, the proposed optimized and compact deep learning system is suitable for mobile-based micro-expression analysis to detect the genuine human emotions.

Cross-Database Micro-Expression Recognition: A Benchmark

Cross-database micro-expression recognition (CDMER) is one of recently emerging and interesting problem in micro-expression analysis. CDMER is more challenging than the conventional micro-expression recognition (MER), because the training and testing samples in CDMER come from different micro-expression databases, resulting in inconsistency of the feature distributions between the training and testing sets. In this paper, we contribute to this topic from three aspects. First, we establish a CDMER experimental evaluation protocol aiming to allow the researchers to conveniently work on this topic and evaluate their proposed methods under the same standard. Second, we conduct benchmark experiments by using NINE state-of-the-art domain adaptation (DA) methods and SIX popular spatiotemporal descriptors for investigating CDMER problem from two different perspectives. Third, we propose a novel DA method called region selective transfer regression (RSTR) to deal with the CDMER task. The overall superior performance of RSTR over the state-of-the-art DA methods demonstrates that taking into consideration the facial local region information used in RSTR contributes to developing effective DA methods for dealing with CDMER problem.

CAS(ME)3: A Third Generation Facial Spontaneous Micro-Expression Database With Depth Information and High Ecological Validity.

Micro-expression (ME) is a significant non-verbal communication clue that reveals one person's genuine emotional state. The development of micro-expression analysis (MEA) has just gained attention in the last decade. However, the small sample size problem constrains the use of deep learning on MEA. Besides, ME samples distribute in six different databases, leading to database bias. Moreover, the ME database development is complicated. In this article, we introduce a large-scale spontaneous ME database: CAS(ME) 3. The contribution of this article is summarized as follows: (1) CAS(ME) 3 offers around 80 hours of videos with over 8,000,000 frames, including manually labeled 1,109 MEs and 3,490 macro-expressions. Such a large sample size allows effective MEA method validation while avoiding database bias. (2) Inspired by psychological experiments, CAS(ME) 3 provides the depth information as an additional modality unprecedentedly, contributing to multi-modal MEA. (3) For the first time, CAS(ME) 3 elicits ME with high ecological validity using the mock crime paradigm, along with physiological and voice signals, contributing to practical MEA. (4) Besides, CAS(ME) 3 provides 1,508 unlabeled videos with more than 4,000,000 frames, i.e., a data platform for unsupervised MEA methods. (5) Finally, we demonstrate the effectiveness of depth information by the proposed depth flow algorithm and RGB-D information.

Capturing Micro-Expressions on Zoom: A Promising Sales Opportunity

The subtleties of salesmanship are changing in this virtual world of online communication. This proof-of￾concept study examines the feasibility of capturing customer images at the two, ten, and thirty-second intervals following a Zoom platform's sales query. A captured image underwent a micro-expression analysis employing a computer-driven program that generates an emotional-algorithm analyzing the emotions of happy, neutral, sad, disgust, fear, surprise, and anger. This algorithm enabled researchers to predict a purchase-decision using proprietary artificial intelligence software. Findings suggest that micro￾expressions captured at the two-second interval exhibited a significant relationship with a customer's purchase decision. A summary table provides a detailed overview of all results using the acronym [MICRO].

Filter selection for image processing before the landmark detection stage for micro-expression analysis

The automated estimation of the psycho-emotional state of the human and their emotional reactions to the different influences using the video image analysis is an urgent task in different fields, such as: safeguarding in manufacturing, aviation, transportation, prevention of the crimes and terroristic threats, marketing researches etc. A promising direction is the facial micro-expression analysis. The facial micro-expressions are not under conscious control and reflect the objective emotional reaction. One of the key stages of the procedure of the automatic emotion estimation by the facial micro-expressions is the correct facial landmark detection. It is a complex task because of the presence of the different noise in the consecutive frames. Purpose – the investigation of the ways of increasing the performance of the facial micro-expression analysis pipeline by using preliminary video image processing procedures. It is shown that, as the preliminary stage of the micro-expression analysis pipeline, it is reasonable to perform the blurring of the original images to obtain the more stable results. The determined filtering parameters provide the MediaPipe framework a performance increase for the micro-expression analysis problems. It is shown that the video image blurring by the Gaussian filter with a size of 15×15 pixels allows to reduce the noise influence and to decrease the incorrect shifts of the facial landmarks from frame to frame induced by this noise. The proposed procedure of preliminary video image processing can be used for increasing the facial micro-expression recognition performance in emotion recognition systems based on the video sequence analysis.

Video-Based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms.

Unlike the conventional facial expressions, micro-expressions are involuntary and transient facial expressions capable of revealing the genuine emotions that people attempt to hide. Therefore, they can provide important information in a broad range of applications such as lie detection, criminal detection, etc. Since micro-expressions are transient and of low intensity, however, their detection and recognition is difficult and relies heavily on expert experiences. Due to its intrinsic particularity and complexity, video-based micro-expression analysis is attractive but challenging, and has recently become an active area of research. Although there have been numerous developments in this area, thus far there has been no comprehensive survey that provides researchers with a systematic overview of these developments with a unified evaluation. Accordingly, in this survey paper, we first highlight the key differences between macro- and micro-expressions, then use these differences to guide our research survey of video-based micro-expression analysis in a cascaded structure, encompassing the neuropsychological basis, datasets, features, spotting algorithms, recognition algorithms, applications and evaluation of state-of-the-art approaches. For each aspect, the basic techniques, advanced developments and major challenges are addressed and discussed. Furthermore, after considering the limitations of existing micro-expression datasets, we present and release a new dataset - called micro-and-macro expression warehouse (MMEW) - containing more video samples and more labeled emotion types. We then perform a unified comparison of representative methods on CAS(ME) 2 for spotting, and on MMEW and SAMM for recognition, respectively. Finally, some potential future research directions are explored and outlined.

MESNet: A Convolutional Neural Network for Spotting Multi-Scale Micro-Expression Intervals in Long Videos.

Micro-expression spotting is a fundamental step in the micro-expression analysis. This paper proposes a novel network based convolutional neural network (CNN) for spotting multi-scale spontaneous micro-expression intervals in long videos. We named the network as Micro-Expression Spotting Network (MESNet). It is composed of three modules. The first module is a 2+1D Spatiotemporal Convolutional Network, which uses 2D convolution to extract spatial features and 1D convolution to extract temporal features. The second module is a Clip Proposal Network, which gives some proposed micro-expression clips. The last module is a Classification Regression Network, which classifies the proposed clips to micro-expression or not, and further regresses their temporal boundaries. We also propose a novel evaluation metric for spotting micro-expression. Extensive experiments have been conducted on the two long video datasets: CAS(ME)2 and SAMM, and the leave-one-subject-out cross-validation is used to evaluate the spotting performance. Results show that the proposed MESNet effectively enhances the F1-score metric. And comparative results show the proposed MESNet has achieved a good performance, which outperforms other state-of-the-art methods, especially in the SAMM dataset.

Rank-Pooling-Based Features on Localized Regions for Automatic Micro-Expression Recognition

Facial micro-expression is a subtle and involuntary facial expression that exhibits short duration and low intensity where hidden feelings can be disclosed. The field of micro-expression analysis has been receiving substantial awareness due to its potential values in a wide variety of practical applications. A number of studies have proposed sophisticated hand-crafted feature representations in order to leverage the task of automatic micro-expression recognition. This paper employs a dynamic image computation method for feature extraction so that features can be learned on certain localized facial regions along with deep convolutional networks to identify micro-expressions presented in the extracted dynamic images. The proposed framework is simple as opposed to other existing frameworks which used complex hand-crafted feature descriptors. For performance evaluation, the framework is tested on three publicly available databases, as well as on the integrated database in which individual databases are merged into a data pool. Impressive results from the series of experimental work show that the technique is promising in recognizing micro-expressions.