STA-GCN: Spatio-Temporal AU Graph Convolution Network for Facial Micro-expression Recognition

Abstract

Facial micro-expression (FME) is a fast and subtle facial muscle movement that typically reflects person’s real mental state. It is a huge challenge in the FME recognition task due to the low intensity and short duration. FME can be decomposed into a combination of facial muscle action units (AU), and analyzing the correlation between AUs is a solution for FME recognition. In this paper, we propose a framework called spatio-temporal AU graph convolutional network (STA-GCN) for FME recognition. Firstly, pre-divided AU-related regions are input into the 3D CNN, and inter-frame relations are encoded by inserting a Non-Local module for focusing on apex information. Moreover, to obtain the inter-AU dependencies, we construct separate graphs of their spatial relationships and activation probabilities. The relationship feature we obtain from the graph convolution network (GCN) are used to activate on the full-face features. Our proposed algorithm achieves state-of-the-art accuracy of 76.08% accuracy and F1-score of 70.96% on the CASME II dataset, which outperformance all baselines.