Micro-expression recognition based on deep capsule adversarial domain adaptation network

Abstract

Micro-expression (ME), which reveals the genuine feelings and motives within human beings, attracts considerable attention in the field of automatic affective recognition. The main challenges for robust micro-expression recognition (MER) are from the short ME duration, low intensity of facial muscle movements, and insufficient samples. To meet these challenges, we propose an optical flow-based deep capsule adversarial domain adaptation network (DCADAN) for MER, which leverages a deep neural network stemming from these speculations. To alleviate the negative impact of the identity related features, optical flow preprocessing is applied to encode the subtle face motion information that is highly related to facial MEs. Then, a deep capsule network is developed to determine the part–whole relationships on optical flow features. To cope with the data deficiency and enhance the generalization capability via domain adaptation, an adversarial discriminator module that enriches the available samples from macro-expression data is integrated into the capsule network to train an expeditious end-to-end deep network. Finally, a simple and yet efficient attention module is embedded to the DCADAN to adaptively aggregate optical flow convolution maps into the primary capsule layers. We evaluate the performance of the entire network on the cross-database ME benchmark (3DB) using the leave-one-subject-out cross-validation. Unweighted F1-score (UF1) and unweighted average recall (UAR) are exploited as the evaluation metrics. The MER based on DCADAN achieves a UF1 score of 0.801 and a UAR score of 0.829 in comparison with a UF1 of 0.788 and a UAR of 0.782 for the updated approach. The comprehensive experimental results show that the incorporation of adversarial domain adaption into the capsule network is feasible and effective for representing discriminative features in ME and the proposed model outperforms state-of-the-art deep learning networks for MER.