Evaluation metrics for video captioning: A survey

Abstract

Automatic evaluation metrics play an important role in assessing video captioning systems. Popular metrics used for assessing such approaches are based on word matching and may fail to evaluate the quality of automatically generated captions due to inherent natural language ambiguity. Moreover, they require many reference sentences for effective scoring. With the fast development of image and video captioning methodologies using deep learning in recent years, many metrics have been proposed for evaluating such approaches. In this study, we present a survey of automatic evaluation metrics for the video captioning task. Moreover, we highlight the challenges in evaluating video captioning and propose a taxonomy to organize the existing evaluation metrics. We also briefly describe and identify the advantages and shortcomings of those metrics and identify applications or contexts in which these metrics can be better used. To identify the advantages and limitations of the evaluation metrics, we quantitatively compare them using videos from different datasets employed for the video description task. Finally, we discuss the advantages and limitations of the metrics and propose some promising future research directions, such as semantic measurement, explainability, adaptability, extension to other languages, dataset limitations, and multimodal free-reference metrics.