Bidirectional image-sentence retrieval by local and global deep matching

Abstract

In this paper, we propose a novel local and global deep matching model to tackle bidirectional image-sentence retrieval. Our proposed matching model can simultaneously exploit the image representation, sentence representation, as well as their complicated matching relationships from both local and global perspectives. For images, two different convolutional neural networks (CNNs) are leveraged to encode the local and global contents, with selective attentions to the image sub-regions and the whole image. For sentences, a CNN based sentence model and Fisher vector are employed to capture the global and local semantic meanings, respectively. Relying on the local and global representations of the image and sentence, the proposed deep matching model learns the complicated image-sentence matching relationships from local and global perspectives by integrating cross-modality correlations with intra-modality similarities. Extensive experimental results demonstrate that the proposed local and global matching model outperforms the state-of-the-art bidirectional retrieval approaches on the Flickr8K, Flickr30K, and MSCOCO datasets. Moreover, the image and sentence representations exploited in local and global levels are demonstrated to play synergic and complementary roles for bidirectional image-sentence retrieval.