Hierarchical Soft Quantization for Skeleton-Based Human Action Recognition

Abstract

In daily life, human beings rely on hands and body parts to complete particular actions cooperatively. These selected body parts and their cooperative relationships are essential cues to distinguish these actions. However, most existing action recognition methods, which try to model the body appearance or spatial relations in skeleton sequences, often ignore the essential cooperation relationship among joints. Differently, in this paper, we propose a spatio-temporal hierarchical soft quantization method to extract the congenerous motion features, which reflect the cooperation relations among joints and body parts. Specifically, we design a hierarchical network with multiple soft quantization layers to extract congenerous features. The hierarchical network not only models the spatial hierarchy of skeleton structure for joint, part, and body, but also extracts the temporal hierarchy with sliding windows for frame, fragment, and sequence. Moreover, the features in each layer are visually explainable, which reflect the cooperation among body parts. The trainable parameters in the network are also significantly reduced, which reduces computational cost. Extensive experiments conducted on four benchmarks demonstrate that our method can provide competitive results compared with state-of-the-arts. The visualized congenerous features also validate that our approach can effectively perceive the essential cooperation relations.