Real-Time Action Detection Method based on Multi-Scale Spatiotemporal Feature

Abstract

Spatiotemporal action detection relies on the learning of video spatial and temporal information. The current state-of-the-art convolutional neural network-based action detectors have achieved remarkable results using 2D CNN or 3D CNN architectures. However, due to the complexity of the network structure and spatiotemporal information perception, these methods are usually used in a non-real-time, offline manner. The main challenge of spatiotemporal action detection is to design an effective detection network architecture and effectively perceive the fused spatiotemporal features. Aiming at the above problems, our paper proposes a real-time action detection method based on multi-scale spatiotemporal feature. Aiming at the problem that only 2D or 3D backbone network cannot effectively model spatiotemporal features, we extract spatiotemporal features by multi-branch feature extraction networks respectively. For the lack of descriptiveness of single-scale spatiotemporal features, a multi-scale spatiotemporal feature-aware attention network is proposed to learn long-term temporal dependencies and spatial context information. And the fusion between temporal and spatial features is guided by fusion attention to highlight more discriminative spatiotemporal feature representations. The proposed method achieves 82.59% and 78.30% accuracy on two spatiotemporal action datasets UCF101-24 and JHMDB-21, respectively and reaching 73 frames/s.