Micro-expression Sequences Research Articles

Counterfactual discriminative micro-expression recognition

Micro-expressions are spontaneous, rapid and subtle facial movements that can hardly be suppressed or fabricated. Micro-expression recognition (MER) is one of the most challenging topics in affective computing. It aims to recognize subtle facial movements which are quite difficult for humans to perceive in a fleeting period. Recently, many deep learning-based MER methods have been developed. However, how to effectively capture subtle temporal variations for robust MER still perplexes us. We propose a counterfactual discriminative micro-expression recognition (CoDER) method to effectively learn the slight temporal variations for video-based MER. To explicitly capture the causality from temporal dynamics hidden in the micro-expression (ME) sequence, we propose ME counterfactual reasoning by comparing the effects of the facts w.r.t. original ME sequences and the counterfactuals w.r.t. counterfactually-revised ME sequences, and then perform causality-aware prediction to encourage the model to learn those latent ME temporal cues. Extensive experiments on four widely-used ME databases demonstrate the effectiveness of CoDER, which results in comparable and superior MER performance compared with that of the state-of-the-art methods. The visualization results show that CoDER successfully perceives the meaningful temporal variations in sequential faces.

Dual-Branch Cross-Attention Network for Micro-Expression Recognition with Transformer Variants

A micro-expression (ME), as a spontaneous facial expression, usually occurs instantaneously and is difficult to disguise after an emotion-evoking event. Numerous convolutional neural network (CNN)-based models have been widely explored to recognize MEs for their strong local feature representation ability on images. However, the main drawback of the current methods is their inability to fully extracting holistic contextual information from ME images. To achieve efficient ME learning representation from diverse perspectives, this paper uses Transformer variants as the main backbone and the dual-branch architecture as the main framework to extract meaningful multi-modal contextual features for ME recognition (MER). The first branch leverages an optical flow operator to facilitate the motion information extraction between ME sequences, and the corresponding optical flow maps are fed into the Swin Transformer to acquire motion–spatial representation. The second branch directly sends the apex frame in one ME clip to Mobile ViT (Vision Transformer), which can capture the local–global features of MEs. More importantly, to achieve the optimal feature stream fusion, a CAB (cross attention block) is designed to interact the feature extracted by each branch for adaptive learning fusion. The extensive experimental comparisons on three publicly available ME benchmarks show that the proposed method outperforms the existing MER methods and achieves an accuracy of 81.6% on the combined database.

Micro Expression Recognition via Dual-Stream Spatiotemporal Attention Network.

Microexpression can manifest the real mood of humans, which has been widely concerned in clinical diagnosis and depression analysis. To solve the problem of missing discriminative spatiotemporal features in a small data set caused by the short duration and subtle movement changes of microexpression, we present a dual-stream spatiotemporal attention network (DSTAN) that integrates dual-stream spatiotemporal network and attention mechanism to capture the deformation features and spatiotemporal features of microexpression in the case of small samples. The Spatiotemporal networks in DSTAN are based on two lightweight networks, namely, the spatiotemporal appearance network (STAN) learning the appearance features from the microexpression sequences and the spatiotemporal motion network (STMN) learning the motion features from optical flow sequences. To focus on the discriminative motion areas of microexpression, we construct a novel attention mechanism for the spatial model of STAN and STMN, including a multiscale kernel spatial attention mechanism and global dual-pool channel attention mechanism. To obtain the importance of each frame in the microexpression sequence, we design a temporal attention mechanism for the temporal model of STAN and STMN to form spatiotemporal appearance network-attention (STAN-A) and spatiotemporal motion network-attention (STMN-A), which can adaptively perform dynamic feature refinement. Finally, the feature concatenate-SVM method is used to integrate STAN-A and STMN-A to a novel network, DSTAN. The extensive experiments on three small spontaneous microexpression data sets of SMIC, CASME, and CASME II demonstrate the proposed DSTAN can effectively cope with the recognition of microexpressions.

Micro‐expression recognition by two‐stream difference network

Facial micro-expression is a superposition of micro-expression features and identity information of a subject. For the problem of identity information interference in micro-expression recognition, this study proposes a new method for facial micro-expression recognition by de-identity information, called two-stream difference network (TSDN). First, a two-stream encoder-decoder network is trained by a convolutional neural network, where the input of the micro-expression stream is a micro-expression image, and the identity stream is a facial identity image. The micro-expression image is the apex image, and the identity image is the onset image in the micro-expression sequence. The identity information and micro-expression features are recorded in the intermediate layer of the micro-expression stream, while the intermediate layer of the identity stream contains only the identity information of a subject. Then, the identity information is removed by the difference network, but micro-expression features are stored in the intermediate layer of the micro-expression stream. Given the sequence of the micro-expressions, the TSDN model of de-identity information learns the difference that stores in the expression stream. Two public spontaneous facial micro-expression data sets (SMIC and CASME II) are employed in our experiments. The experiment results show that our model can achieve a superior performance in micro-expression recognition.

Enhanced Forest Microexpression Recognition Based on Optical Flow Direction Histogram and Deep Multiview Network

In order to recognize the instantaneous changes of facial microexpressions in natural environment, a method based on optical flow direction histogram and depth multiview network to enhance forest microexpression recognition was proposed. In the preprocessing stage, the histogram equalization of the acquired face image is performed, and then the dense key points of the face are detected. According to the coordinates of the key points and the face action coding system (FACS), the face region is divided into 15 regions of interest (ROI). In the feature extraction stage, the optical flow direction histogram feature between adjacent frames in ROI is extracted to detect the peak frame of microexpression sequence. Finally, the average optical flow direction histogram feature of the image sequence from the initial frame to the peak frame is extracted. In the classification stage, firstly, the head pose parameters under horizontal degrees of freedom are estimated to eliminate the influence of head pose motion, and a forest multiview conditional probability model based on deep multiview network is established. Conditional probability and neural connection function are introduced into the node splitting learning of random tree to improve the learning ability and distinguishing ability of the model on the limited training set. Finally, multiview-weighted voting is used to determine the categories of facial microexpressions. Experiments on CASME II microexpression dataset show that the proposed method can effectively describe the changes of microexpressions and improve the recognition accuracy compared with other new methods.

Mean oriented Riesz features for micro expression classification

Micro-expressions are brief and subtle facial expressions that go on and off the face in a fraction of a second. This kind of facial expressions usually occurs in high stake situations and is considered to reflect a human’s real intent. There has been some interest in micro-expression analysis, however, a great majority of the methods are based on classically established computer vision methods such as local binary patterns, histogram of gradients and optical flow. A novel methodology for micro-expression recognition using the Riesz pyramid, a multi-scale steerable Hilbert transform is presented. In fact, an image sequence is transformed with this tool, then the image phase variations are extracted and filtered as proxies for motion. Furthermore, the dominant orientation constancy from the Riesz transform is exploited to average the micro-expression sequence into an image pair. Based on that, the Mean Oriented Riesz Feature description is introduced. Finally the performance of our methods are tested in two spontaneous micro-expressions databases and compared to state-of-the-art methods.

Spatiotemporal Recurrent Convolutional Networks for Recognizing Spontaneous Micro-Expressions

Recently, the recognition task of spontaneous facial micro-expressions has attracted much attention with its various real-world applications. Plenty of handcrafted or learned features have been employed for a variety of classifiers and achieved promising performances for recognizing micro-expressions. However, the micro-expression recognition is still challenging due to the subtle spatiotemporal changes of micro-expressions. To exploit the merits of deep learning, we propose a novel deep recurrent convolutional networks based micro-expression recognition approach, capturing the spatial-temporal deformations of micro-expression sequence. Specifically, the proposed deep model is constituted of several recurrent convolutional layers for extracting visual features and a classificatory layer for recognition. It is optimized by an end-to-end manner and obviates manual feature design. To handle sequential data, we exploit two types of extending the connectivity of convolutional networks across temporal domain, in which the spatiotemporal deformations are modeled in views of facial appearance and geometry separately. Besides, to overcome the shortcomings of limited and imbalanced training samples, temporal data augmentation strategies as well as a balanced loss are jointly used for our deep network. By performing the experiments on three spontaneous micro-expression datasets, we verify the effectiveness of our proposed micro-expression recognition approach compared to the state-of-the-art methods.

MERTA: micro-expression recognition with ternary attentions

Micro-expression is a spontaneous and uncontrollable way to convey emotions. It contains abundant psychological information, whose recognition has significant importance in various fields. In recent years, with the rapid development of computer vision, the research of facial expression tends to be more mature while the research of micro-expression remains a hot yet challenging topic. The main difficulties of recognizing micro-expression lay on the discriminative feature extraction process due to the extremely short-term and subtlety of micro-expression. To deal with this problem, this paper proposes a deep learning model to efficiently extract discriminative features. Our model is based on three VGGNets and one Long Short-Term Memory (LSTM). Three VGGNets are used to extract static and motive information where three types of attention mechanism are jointly integrated for more discriminative visual representations. Then, the spatial features of a micro-expression sequence are sequentially fed into an LSTM to extract spatio-temporal features and predict the micro-expression category. Our algorithm is carried out on the benchmark micro-expression dataset CASME II. Its efficiency is demonstrated by extensive ablation analysis and state-of-the-art algorithms.

Facial microexpression recognition based on adaptive key frame representation

Recognizing microexpression serves as a vital clue for affective estimation. Fast and discriminative feature extraction has always been a critical issue for spontaneous microexpression recognition applications. A microexpression analysis framework is proposed by adaptively key frame extraction and representation. First, to remove redundant information in the microexpression video sequences, the key frame is adaptively selected on the criteria of structural similarity index between different face images, Second, robust principal component analysis is applied to obtain the sparse information in the key frame, which not only retains the expression attributes of the microexpression sequence, but also eliminates useless interference. Furthermore, we construct dual-cross patterns to get the final microexpressions representation for classification. Repeated comparison experiments were performed on the SMIC and CASME2 databases to evaluate the performance of the proposed method. Experimental results demonstrate that our proposed method gets higher recognition rates and achieves promising performance, compared with the traditional microexpression recognition.

An Improved Micro-Expression Recognition Method Based on Necessary Morphological Patches

Micro-expression is a spontaneous emotional representation that is not controlled by logic. A micro-expression is both transitory (short duration) and subtle (small intensity), so it is difficult to detect in people. Micro-expression detection is widely used in the fields of psychological analysis, criminal justice and human-computer interaction. Additionally, like traditional facial expressions, micro-expressions also have local muscle movement. Psychologists have shown micro-expressions have necessary morphological patches (NMPs), which are triggered by emotion. Furthermore, the objective of this paper is to sort and filter these NMPs and extract features from NMPs to train classifiers to recognize micro-expressions. Firstly, we use the optical flow method to compare the on-set frame and the apex frame of the micro-expression sequences. By doing this, we could find facial active patches. Secondly, to find the NMPs of micro-expressions, this study calculates the local binary pattern from three orthogonal planes (LBP-TOP) operators and cascades them with optical flow histograms to form the fusion features of the active patches. Finally, a random forest feature selection (RFFS) algorithm is used to identify the NMPs and to characterize them via support vector machine (SVM) classifier. We evaluated the proposed method on two popular publicly available databases: CASME II and SMIC. Results show that NMPs are statistically determined and contribute to significant discriminant ability instead of holistic utilization of all facial regions.

Spatiotemporal Feature Descriptor for Micro-Expression Recognition Using Local Cube Binary Pattern

Micro-expression recognition has been an active research area in recent years, it plays an important role in psychology and public security. Due to the aspects of short duration and subtle movement, it is challenging to extract spatiotemporal features of micro-expressions. The existing methods only extract features in the three-dimensional orthogonal plane and fail to make full use of that information. To solve this problem, we propose a new Local Cubes Binary Patterns (LCBP) method for micro-expression recognition. LCBP is cascaded by the motion information LCBP direction , the amplitude information LCBP amplitudes , and the spatial information LCBP3D to obtain the spatiotemporal features. The advantage of LCBP is its ability to preserve the spatiotemporal information and the low feature dimension. Furthermore, to increase the discrimination of features in micro-expression sequences, we apply a differential calculation energy map to find regions of interest (ROI) for getting a weighted energy map. The final micro-expression feature acquired by fusing the LCBP features and the weighted energy map are classified through the Support Vector Machine (SVM). We evaluate the proposed method on four published micro-expression databases including SMIC, CASME, CASME2, SAMM. Experimental results demonstrate that our proposed method achieves promising performance for micro-expression recognition.

Microexpression Identification and Categorization Using a Facial Dynamics Map

Unlike conventional facial expressions, microexpressions are instantaneous and involuntary reflections of human emotion. Because microexpressions are fleeting, lasting only a few frames within a video sequence, they are difficult to perceive and interpret correctly, and they are highly challenging to identify and categorize automatically. Existing recognition methods are often ineffective at handling subtle face displacements, which can be prevalent in typical microexpression applications due to the constant movements of the individuals being observed. To address this problem, a novel method called the Facial Dynamics Map is proposed to characterize the movements of a microexpression in different granularity. Specifically, an algorithm based on optical flow estimation is used to perform pixel-level alignment for microexpression sequences. Each expression sequence is then divided into spatiotemporal cuboids in the chosen granularity. We also present an iterative optimal strategy to calculate the principal optical flow direction of each cuboid for better representation of the local facial dynamics. With these principal directions, the resulting Facial Dynamics Map can characterize a microexpression sequence. Finally, a classifier is developed to identify the presence of microexpressions and to categorize different types. Experimental results on four benchmark datasets demonstrate higher recognition performance and improved interpretability.

Micro-expression recognition based on CBP-TOP feature with ELM

Micro-expressions are short, involuntary facial expressions with revealing suppressed effect that people try to conceal. Detecting and recognizing micro-expressions have potential applications in clinical and national security fields. Centralized Binary Patterns from Three Orthogonal Panels (CBP-TOP) is a novel approach that can efficiently extract the micro-expression information in spatial and temporal domains. In this method, the micro-expression image sequences are initially preprocessed that includes face detection, interception, size normalization, and micro-expression detection. Then, the features of the human face are extracted from the blocks of images, using CBP-TOP operator. Finally, extreme learning machine (ELM) is used to recognize micro-expressions. The experimental results show that this method can extract motion features and dynamic texture information in micro-expression sequences more efficiently and has greater improvement in micro-expressions recognition rate than the traditional recognition methods.