Adaptive Attention Mechanism Based Semantic Compositional Network for Video Captioning

Abstract

Video captioning task is to generate a text to describe the content in the video. To generate a proper description, many people have begun to add explicit semantic information to the video generation process. However, in recent work, with the mining of semantics in video, the semantic information in some existing methods will play a smaller and smaller role in the decoding process. Besides, decoders apply temporal attention mechanisms to all generation words including visual vocabulary and non visual vocabulary that will produce inaccurate or even wrong results. To overcome the limitations, 1) we detect visual feature to composite semantic tags from each video frame and introduce a semantic combination network in the decoding stage. We use the probability of each semantic object as an additional parameter in the long-short term memory(LSTM), so as to better play the role of semantic tags, 2) we combine two levels of LSTM with temporal attention mechanism and adaptive attention mechanism respectively. Then we propose an adaptive attention mechanism based semantic compositional network (AASCNet) for video captioning. Specifically, the framework uses temporal attention mechanism to select specific visual features to predict the next word, and the adaptive attention mechanism to determine whether it depends on visual features or context information. Extensive experiments conducted on the MSVD video captioning dataset prove the effectiveness of our method compared with state-of-the-art approaches.