Leveraging unpaired out-of-domain data for image captioning

Abstract

Image captioning is a focal point in the realm of computer vision due to its scientific and practical values. Although recent encoder-decoder models have achieved promising performance, they only leverage data from standard datasets, and the performance is limited to the specific datasets. There’re still large amounts of data without any additional annotations on the internet which can’t be fully utilized. In this paper, we propose a novel approach to using external unpaired images and texts to enhance the performance of image captioning system. Our method can utilize image and text data scraped from the internet respectively to improve the performance limited in concepts-decoder framework. Our approach can transfer the knowledge learned from web data to the standard dataset. We conduct extensive experiments on MS COCO and Flickr30K datasets. The result demonstrates the effectiveness of our method. On both datasets, our metrics scores are significantly improved compared with other methods.