Recurrent convolutional video captioning with global and local attention

Abstract

Video captioning with encoder–decoder structures has been extensively studied in the recent literature, where a great deal of work focuses on multimodal features and attention mechanisms. Most of the previous work uses only the global temporal features, such as image, motion, and audio features, and ignores the local semantic features existing extensively in the video data. Furthermore, it is difficult to fully utilize the local features due to frame-to-frame redundancy. In this paper, we propose to combine global temporal features and local object-based features in a complementary way to develop a multimodal attention mechanism (global–local attention mechanism). Based on this attention mechanism, we introduce a novel video captioning method, Recurrent Convolutional Video Captioning with Global and Local Attention (RCGL). Further, both LSTM and 1D CNN are incorporated into the decoder to improve the long-range dependency. The experimental results on two standard datasets, MSVD and MSR-VTT, demonstrate that RCGL outperforms the state-of-the-art in four common metrics.