RVAIC: Refined visual attention for improved image captioning

Abstract

Visual attention has emerged as a prominent approach for improving the effectiveness of image captioning, as it enables the decoder network to focus selectively on the most salient regions in the image content, thereby facilitating the generation of precise and informative captions. Although visual attention achieves the improvement, the small numerical values of its input have a negative impact on its softmax, decreasing its effectiveness. To address this limitation, we propose a refined visual attention (RVA) framework that internally reweights visual attention by leveraging the language context of previously generated words. We first feed the language context into a fully connected layer to obtain appropriate dimensions for the visual features. Then, we use a sigmoid function to obtain a probability distribution to reweight the softmax’s input by applying the multiplication process. Experiments conducted on the MS COCO dataset demonstrate that RVA outperforms traditional visual attention and other existing image captioning methods, highlighting its effectiveness in enhancing the accuracy and informativeness of image captions.