Real-Time Micro-expression Detection in Unlabeled Long Videos Using Optical Flow and LSTM Neural Network

Abstract

Micro-expressions are momentary involuntary facial expressions which may expose a person’s true emotions. Previous work in micro-expression detection mainly focus on finding the peak frame from a video sequence that has been determined to have a micro-expression, and the amount of computation is usually very large. In this paper, we propose a real-time micro-expression detection method based on optical flow and Long Short-term Memory (LSTM) to detect the appearance of micro-expression. This method takes only one step of data preprocessing which is less than previous work. Specifically, we use a sliding window with fixed-length to split a long video into several short videos, then a new and improved optical flow algorithm with low computational complexity was developed to extract feature curves based on the Facial Action Coding System (FACS). Finally, the feature curves were passed to a LSTM model to predict whether micro-expression occurs. We evaluate our method on CASMEll and SAMM databases, and it achieves a new state-of-the-art accuracy (89.87%) on CASMEll database (4.54% improvement). Meanwhile our method only takes 1.48 s to detect the micro-expression in a video sequence with 41 frames (the frame rate is about 28fps). The experimental results show that the proposed method can achieve better comprehensive performances.