TRACS - Transformer for Video Captioning and Summarisation.pdf

Abstract

Video summarisation is the process of creating a meaningful and self-explanatory summary of a given video by automatically selecting keyframes and thus creating a short and concise video summary of the original video clip. Video captioning refers to the task of automatically generating natural language description of a given video to provide some additional or interpretive information. Video summarisation and video captioning are often considered as two different tasks, each with a diverse application field. Thus, we propose an approach to jointly combine these two tasks and present a model which generates a short video summary along with a relevant caption. We make use of self-attention based transformer network combined with Multi Layer Perceptron (MLP) with multiple hidden layers for video captioning and video summarisation. We aim to demonstrate that the joint model can attain better performance than many of the previous approaches in both of the individual tasks. Our proposed model is based on the fact that both of the above mentioned tasks of captioning and summarisation will improve the performance of the other one as both the tasks are complementary to each other