Abstract
• The Attention-based Feature Decoupling module is proposed to obtain AU-specific spatio-temporal feature vectors. • The Co-occurrence Knowledge Graph module is proposed to introduce the prior spatio-temporal co-occurrence between AUs. • The Spatio-Temporal Transformer is proposed to obtain the spatio-temporal cooccurrence pattern in a data-driven manner. • An end-to-end facial AU detection network based on heterogeneous spatiotemporal relation learning is proposed. Properties such as temporal relation and action relation of facial Action Units (AUs) make AU detection different from general multi-label classification tasks. Therefore, how to capture the spatial and temporal co-occurrence of AUs becomes the key to improving detection accuracy. Although many works on the spatial relation of AUs have been proposed in recent years, very few works have explored the temporal relation of AUs. In this paper, we propose a Heterogeneous Spatio-Temporal Relation learning network (HSTR-Net) to capture the temporal and spatial relations of AUs. The co-occurrence knowledge graph module guides the network to model spatio-temporal relations by introducing prior relation information, and the spatio-temporal Transformer module adaptively captures spatio-temporal relations through spatio-temporal feature interaction. To further model the temporal relation between AUs, the self-attention mechanism to fuse AU features in the time dimension. Extensive experiments are conducted on two challenging datasets, BP4D and DISFA, and experimental results show that our proposed HSTR-Net achieves comparable performance of the state-of-the-art in the field of AU detection. The code for our methods is available at https://github.com/saaaaalmon/HSTR-Net .
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have