Abstract
Video captioning is a task of automatically describing visual content of a video with a sentence. Recent works in video captioning focus on improving the performance of sentence accuracy. However, the distinctiveness of sentence, i.e., highlighting unique and accurate details of video, in video captioning is still underexplored. This paper aims to improve sentence distinctiveness by incorporating video retrieval into the training process of the video captioning model. Specifically, the video retrieval will calculate similarity scores between the input text generated by the video captioning and videos. These similarity scores are then incorporated into the training loss of video captioning, which serves as distinctiveness constraint where the generated sentence and its corresponding video should have the highest similarity scores. To further improve the sentence distinctiveness, we additionally use reference scores, i.e., similarity scores between ground truth sentences and videos, as weights to scale the training loss of video captioning. This reference score serves as a target score for the model, indicating the desired level of distinctiveness for the generated sentence on how similar the generated sentence should be to the ground truth sentence for the corresponding video. Our qualitative and quantitative results show that our method improves sentence distinctiveness while simultaneously increasing its accuracy on MSVD and MSR-VTT datasets.
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.