Diverse local facial behaviors learning from enhanced expression flow for microexpression recognition

Abstract

Facial microexpression recognition (FMER) has recently gained increasing attention in the interrogation and clinical diagnosis fields. However, accurate FMER suffers from not only the low-intensity muscle movements and a short duration of microexpressions but also limited training data. To address the abovementioned issues, this paper employs facial u/v/s images to explore short durationlow-intensity muscle movements, which can be extracted from the expression flow. Specifically, we propose a motion detail enhancement method to disentangle microexpression-irrelevant motions from the expression flow. Then, a novel data augmentation strategy based on different u-v weights is introduced to generate diverse s components, resolving the limited scale issue. Afterward, the enhanced facial u/v/s images are fed to a novel local-diverse facial microexpression recognition network (LD-FMERN) for microexpression-related feature extraction, where a spatial-channel modulator is used to refine the extracted features. A locally diverse feature mining strategy further enhances the refined features, forcing the network to focus on small and diverse facial regions. Following the law of human vision, we propose an adaptive loss function, which comprises a supervised cross-entropy loss and a self-supervised local-diverse loss, to optimize the network. Extensive quantitative and qualitative evaluations on benchmark datasets, CASMEII, SAMM, and MMEW, demonstrate the improvements. Comparisons with state-of-the-art FMER methods reveal the superiority of the proposed method.