Short Video Representation Learning Based on Convolution Network with Text Attention Mechanism

Abstract

Learning video representation with convolution network methods has been proven useful to learn video representation across many video understanding tasks, including video classification, action recognition and object detection. However, traditional convolution methods fail to take the text feature of the video into consideration, which may contain large amount of useful information for final video representation. Besides, current short video often contains a lot of human language from the start to the end. In this paper, we propose a novel neural network system that learns video representation by aggregating information from both video features and text features of the video. This network, named Conv-Text Net, can effectively learn video representation with its text information by a text attention mechanism. Results show that Conv-Text Net achieves competitive performance in video classification task on Short-Video datasets. We provide analysis towards the behavior of our model and show its robustness to errors in proposals. Experiment shows that our method performs well on video classification task and achieves good score on Short Video data set.