AP-GAN: Predicting skeletal activity to improve early activity recognition

Abstract

Early activity recognition is a classification task before the completion of activity. The study of early activity recognition is beneficial to avoid serious result. Previous studies have focused on extracting effective activity features and modeling for quick and accurate classification. It is challenging because of lack of available information. In order to get a firm basis for judgment, this paper adds an activity prediction module prior to recognition module. The main task of the module is to predict subsequent motions according to observed motions. To avoid motion blur, the structure of GAN (Generative Adversarial Networks) is used to generate the predicted motions. Compared with the traditional deep learning model, dilated neural network has advantages in large-span spatiotemporal feature modeling. The dilated RNN (Recurrent Neural Networks) and CNN (Convolutional Neural Networks) are introduced to the recognition module. In order to make the activity prediction and recognition modules work together, this paper designs and introduces a hard class mining mechanism to improve the learning ability of hard class samples. The proposed method is validated on four skeletal activity datasets and achieves state-of-the-art accuracy.