A convolutional autoencoder model with weighted multi-scale attention modules for 3D skeleton-based action recognition

Abstract

The 3D skeleton sequences of action can be recognized based on series of meaningful movements including changes in the direction and geometry features of the body pose. In this paper, we introduce the 3DPo-CDP descriptor, which incorporates the change direction patterns of body joints and pose features in a unified deep structure to learn more adequate features for the action recognition problem. To this end, two types of features are extracted. First, Change Direction Patterns (CDPs) are extracted by following the important points of motion trajectories where a significant change of direction has occurred using two filtering phases. The CDPs capture the global features which are invariant to noise and insignificant temporal dynamics of joints. Second, Pose Features are employed to learn the intrinsic connectivity relationships of adjacent limbs and the variance distances of body joints from representative joints to concentrate on key-frames and informative joints. The complementary features of CDPs and 3D pose, which are transformed into images, are combined in a unified representation and fed into a new convolutional autoencoder. Unlike conventional convolutional autoencoders that focus on frames, high-level discriminative features of spatiotemporal relationships of whole body joints are extracted by introducing weighted multi-scale channel and spatial attention modules. In this paper, we show that adjacent and non-adjacent neighbors can be effectively used to compute different weights for extracting cross-interaction channels and multi-scale spatial relationships of the current pixel. The extracted features are combined with the wavelet representation of statistical body information and then classified with a multi-class SVM classifier. The experimental results demonstrate the effectiveness of the augmented 3DPo-CDP descriptor using an attentional convolution autoencoder structure on five challenging 3D action recognition datasets.