A multimodal fusion approach for image captioning

Abstract

Deep convolution neural networks connected with the recurrent neural networks are potent models that have achieved excellent performance on image caption task. Although many methods based on the neural network can generate fluent and complete sentences, the image feature vectors extracted by the convolution neural network can only retain a few significant features of the original image, which will lose a lot of useful image information. Moreover, RNNs have a gradient vanishing problem with the growth of RNNs time step, and the generation of sentences will lack the guidance of previous information. In this paper, we introduce a multimodal fusion method for generating descriptions to explain the content of images. Our model consists of four sub-networks: a convolutional neural network for image feature extraction, a ATTssd model for image attributes extraction, a language CNN model CNNm for sentence modeling and a recurrent network (e.g., GRU, LSTM, etc.) for word prediction. Compared with existing methods which predict next word based on one previous word and hidden state, our model uses image attributes information to enhance the image representation and handles all the previous words to modeling the long-term dependencies of history words. The methods are evaluated on the Flickr8k, Flickr30k and MSCOCO datasets. We prove that our model combined with ATTssd and CNNm can significantly enhance the performance, and achieve the competitive results.