Skeleton-based action recognition using sparse spatio-temporal GCN with edge effective resistance

Abstract

Graph convolutional neural networks have established significant success in solving various machine learning and computer vision problems. For skeleton-based action recognition, graph convolutional neural networks are the most suitable choice since human skeleton resembles to a graph. Stacking body skeletons over the length of video sequence results in a very complex spatio-temporal graph of many nodes and edges. Modeling the graph convolutional network directly with such a complex graph curtails the performance due to the redundancy of insignificant nodes and edges in the graph. Also for skeleton-based action recognition, the long-term contextual information is of central importance and many current architectures may fail to capture such contextual information. Therefore in order to alleviate these problems, we propose graph sparsification technique using edge effective resistance to better model the global context information and to eliminate redundant nodes and edges in the graph. Furthermore, we incorporate self-attention graph pooling to retain local properties and graph structures while pooling operation. To the best of our knowledge, we are the first to apply graph sparsification using edge effective resistance for skeleton-based action recognition and our proposed method is confirmed to be effective on action recognition, which achieves state-of-the-art results on publicly available datasets: UTD-MHAD, J-HMDB, NTU-RGB + D-60, NTU-RGB + D-120 and Kinetics dataset.