Multi-derivative physical and geometric convolutional embedding networks for skeleton-based action recognition

Abstract

Action involves rich geometric and physical properties hidden in the spatial structure and temporal dynamics. However, there is a lack of synergy in investigating these properties and their joint embedding in the existing literature. In this paper, we propose a multi-derivative physical and geometric embedding network (PGEN) for action recognition from skeleton data. We model the skeleton joint and edge information using multi-derivative physical and geometric features. Then, a physical and geometric embedding network is proposed to learn co-occurrence features from joints and edges, respectively, and construct a unified convolutional embedding space, where the physical and geometric properties can be integrated effectively. Furthermore, we adopt a multi-task learning framework to explore the inter-dependencies between the physical and geometric properties of the action, which significantly improves the discrimination of the learned features. The experiments on the NTU RGB+D, NTU RGB+D 120, and SBU datasets demonstrate the effectiveness of our proposed representation and modeling method.