LGST-Drop: label-guided structural dropout for spatial–temporal convolutional neural networks

Abstract

Region dropout regularization strategies have proven to be highly effective at improving the generalization performance of convolutional neural networks (CNNs) in a variety of computer vision tasks including image classification, object detection, and semantic segmentation because these strategies enable models to focus on a wider range of image region information. However, for action recognition, models need to be able to extract not only useful spatial information but also important temporal and motion information, which cannot be satisfied by traditional regularization strategies. We propose a spatiotemporal dropout strategy to meet the need for regularization in spatial–temporal CNNs. We call it label guided spatial–temporal drop (LGST-Drop); it not only provides effectively structured dropout in the spatial dimension but also regularizes motion information in the temporal dimension. In addition, LGST-Drop’s mask is guided by the predicted categories of the model itself, which we called temporary labels. Extensive experiments on several standard datasets from action recognition domains show the usefulness of the proposed technique in comparison with the previous methods and theirstate-of-the-art variant algorithms.