Relation Network and Causal Reasoning for Image Captioning

Abstract

Image captioning is a cross-modal problem combining computer vision and natural language processing. A typical image captioning model uses a convolutional neural network to extract the features of an image and then uses an Long Short-Term Memory network to transform the representations of the features. However, this method has problems such as not including high-level semantics in the visual network and exposure bias in the language network. To overcome these problems, this paper proposes a novel image captioning model that combines relationship-aware and reinforcement learning. First, we design a relational awareness network as the visual network to mine the latent relationships between objects in an image. Then, a context semantic relational network is proposed to improve the accuracy of image captioning. The context semantic network can generate feature representations for arbitrary pixel positions in an image without association with any specific visual concepts. Subsequently, the high-level context semantics are used as external knowledge to guide the language network in generating sentences. Finally, a policy gradient training algorithm is designed to simplify the state value function in reinforcement learning. We have verified the effectiveness of the model on the MS-COCO and Flickr 30K datasets. The experimental results show that the model proposed in this paper achieves state-of-the-art results.