Abstract
Image data contain spatial information only, thus making two-dimensional (2D) Convolutional Neural Networks (CNN) ideal for solving image classification problems. On the other hand, video data contain both spatial and temporal information that must be simultaneously analyzed to solve action recognition problems. 3D CNNs are successfully used for these tasks, but they suffer from their extensive inherent parameter set. Increasing the network’s depth, as is common among 2D CNNs, and hence increasing the number of trainable parameters does not provide a good trade-off between accuracy and complexity of the 3D CNN. In this work, we propose Pooling Block (PB) as an enhanced pooling operation for optimizing action recognition by 3D CNNs. PB comprises three kernels of different sizes. The three kernels simultaneously sub-sample feature maps, and the outputs are concatenated into a single output vector. We compare our approach with three benchmark 3D CNNs (C3D, I3D, and Asymmetric 3D CNN) and three datasets (HMDB51, UCF101, and Kinetics 400). Our PB method yields significant improvement in 3D CNN performance with a comparatively small increase in the number of trainable parameters. We further investigate (1) the effect of video frame dimension and (2) the effect of the number of video frames on the performance of 3D CNNs using C3D as the benchmark.
Highlights
C ONVOLUTIONAL Neural Networks (CNN) have been studied extensively in the last decade and have become the preferred intelligence modeling algorithm for many computer vision tasks
Image data contain spatial information only, making two-dimensional (2D) Convolutional Neural Networks (CNN) ideal for solving image classification problems. Video data contain both spatial and temporal information that must be simultaneously analyzed to solve action recognition problems. 3D CNNs are successfully used for these tasks, but they suffer from their extensive inherent parameter set
Our objective is to improve the inference accuracy of 3D CNNs without significantly increasing the number of trainable parameters
Summary
C ONVOLUTIONAL Neural Networks (CNN) have been studied extensively in the last decade and have become the preferred intelligence modeling algorithm for many computer vision tasks. CNNs can learn relevant higher-order information from structured data, which is believed to be similar to how the human brain learns. Many CNNs include data normalizing layers to perform what is commonly referred to as batch normalization (BN) [2]. BN significantly reduces the training time of very deep CNNs by reducing internal covariate shifts. Single-stream 2D CNNs have demonstrated exceptional performance in solving image classification (recognition) [3]–[7], object segmentation [8], and object detection problems [9], [10]. A single-stream 2D CNN cannot be applied to action recognition in video data because the recognition of dynamic actions requires the simultaneous analysis of spatial and temporal information, but 2D CNNs can learn either spatial or temporal information at a time
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
