Micro-expression spotting with a novel wavelet convolution magnification network in long videos

Abstract

Facial Micro-Expressions (MEs) are transient and spontaneous, reflecting a person's authentic internal emotions and have more significant value in many fields. Due to the presence of many background disturbances, including irrelevant motion (such as blinks and head movements) and noise in long videos, it is challenging to spot subtle MEs from these disturbances. To spot subtle MEs, a novel Wavelet Convolution Magnification Network (WCMN) with optical flow feature enhancement for spotting facial micro-expressions in long videos is proposed, which has a U-Net-like architecture and consists of discrete wavelet transform networks and an attention magnification mechanism. It can effectively suppress background disturbances and magnify the optical flow features of MEs, making them easy to detect. Experiments are conducted on two long video datasets (CAS(ME)2 and SAMM Long Videos). The results show that our WCMN outperforms the state-of-the-art ME spotting methods, achieving overall F1-scores of 0.3791 on CAS(ME)2, and 0.3272 on SAMM Long Videos, respectively. We also provide visualization and analysis to explain what the network enhances, and to demonstrate the effectiveness of our WCMN through ablation studies.