Improving distinctiveness in video captioning with text-video similarity

Abstract

Video captioning is a task of automatically describing visual content of a video with a sentence. Recent works in video captioning focus on improving the performance of sentence accuracy. However, the distinctiveness of sentence, i.e., highlighting unique and accurate details of video, in video captioning is still underexplored. This paper aims to improve sentence distinctiveness by incorporating video retrieval into the training process of the video captioning model. Specifically, the video retrieval will calculate similarity scores between the input text generated by the video captioning and videos. These similarity scores are then incorporated into the training loss of video captioning, which serves as distinctiveness constraint where the generated sentence and its corresponding video should have the highest similarity scores. To further improve the sentence distinctiveness, we additionally use reference scores, i.e., similarity scores between ground truth sentences and videos, as weights to scale the training loss of video captioning. This reference score serves as a target score for the model, indicating the desired level of distinctiveness for the generated sentence on how similar the generated sentence should be to the ground truth sentence for the corresponding video. Our qualitative and quantitative results show that our method improves sentence distinctiveness while simultaneously increasing its accuracy on MSVD and MSR-VTT datasets.