Real-time spatiotemporal action localization algorithm using improved CNNs architecture

Abstract

This paper aims to propose a faster and more accurate network for human spatiotemporal action localization tasks. Like the YOWO model, we also use convolutional neural networks (CNNs) for feature extraction, but our model differs from YOWO in three significant ways: firstly, we don’t use the feature fusion strategy, we only use spatial features extracted by 2D CNNs for action localization and spatiotemporal features extracted by 3D CNNs for action recognition; secondly, we make an improvement to the 2D CNNs network by introducing a coordinate attention mechanism and utilize the CIoU loss instead of the coordinate offset loss for bounding box regression; thirdly, we provide a more lightweight and faster spatiotemporal action localization architecture, which reduces the number of parameters by 21.76 million and achieves a speed of 39 fps on 16-frame input clips compared to the YOWO model. We test our model’s performance on three public datasets: UCF-Sports, JHMDB-21 and UCF101-24. Compared with the YOWO model, we improve frame-mAP (@IoU 0.5) by 17.09% and 7.15% on the UCF-Sports and JHMDB-21 datasets, and for video-mAP, we improve by 2.7%, 8.7% and 14.4% at IoU thresholds of 0.2, 0.5 and 0.75 on the JHMDB-21 dataset.