An asymmetrical-structure auto-encoder for unsupervised representation learning of skeleton sequences

Abstract

In this paper, we propose a novel framework for unsupervised representation learning using a structure-asymmetrical auto-encoder in which a 2D-CNN-based encoder learns separable spatiotemporal representations in a low-dimensional feature space under the supervision of salient skeleton motion cues. This study addresses the problem of learning action representations of skeleton sequences. The network captures not only correlations of adjacent joints but also long-term motion dependencies by using the proposed unsupervised training, which leads to the advantage that similar movements are gathered around the same cluster, whereas different movements are gathered around distinct clusters. Our method is unsupervised and does not rely on annotations to associate skeleton sequences with actions. Experimental results clearly showed the effectiveness of the proposed representation learning, and improvements compared with skeleton-based generative learning methods. When the proposed network was fine-tuned with partial labeled data, our results still outperformed some fully supervised methods.