BiGRU-RA Model for Image Chinese Captioning via Global and Local Features

Abstract

<p indent=0mm>To address the problem of insufficient detailed semantic information in current global features-based image captioning models, an image Chinese captioning model combining global and local features is proposed. The proposed model adopts the encoder-decoder framework. In the coding stage, the residual networks (ResNet) and Faster R-CNN are used to extract the global and local features of images respectively, improving the model ҆ s utilization of image features at different scales. A bi-directional gated recurrent unit (BiGRU) with embedded visual attention structure and residual connection structure is applied as the decoder (BiGRU with residual connection and attention, BiGRU-RA). The model can adaptively allocate image features and text weights, and improve the mapping relationship between image feature regions and context information. Additionally, the reinforcement learning-based policy gradient is added to improve the loss function of the model and optimize the evaluation criteria CIDEr directly. The training and experiments are conducted on the Chinese captioning dataset of AI challenger. The comparative results show that the proposed model obtained better scores and the generated caption are more accurate and detailed.