Video captioning based on dual learning via multiple reconstruction blocks

Abstract

In the context of video captioning, a conventional dual learning scheme involves two tasks: a primal task, which translates frame features into natural language captions, and a dual task, which reconstructs frame features from the generated captions. The dual task serves as a regularization mechanism for the primal task, providing feedback that helps to improve the accuracy of the generated captions. In prior research, it has been demonstrated that the inclusion of dual learning regularization into the architecture of a video captioning model can substantially improve performance. However, it remains an open question whether the performance of such a model can be further enhanced through the incorporation of additional regularizers. In this study, we investigate the use of multiple blocks of primal and dual tasks as additional regularizers in the model. Our experiments on benchmark datasets show that the appropriate number of additional regularizers can further improve the quality of the video captioning model and achieve state-of-the-art results.