Learning Homogeneous and Heterogeneous Co-Occurrences for Unsupervised Cross-Modal Retrieval

Abstract

Image-text retrieval, which focuses on unifying both visual and textual representations, is one of the major tasks of cross-modal information processing. With attention mechanism, previous methods performing well take advantage of not only the correspondence in image-text level but also the semantic alignment between the regions in images and corresponding words. However, few of them comprehend the importance of combing the semantic relationship between multimodalities and semantic correspondences in one modality at the same time. Inspired by the heterogeneous information learning of heterogeneous graph network, we propose a novel method called Homogeneous and Heterogeneous Co-Occurrences (H <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</inf> CO) which mainly consists of two modules to achieve modal co-occurrence in a query of corporations. Specifically, Homogeneous Co-Occurrence Module captures correspondences with neighbors from single modal of regions and words respectively, while Heterogeneous Co-Occurrence Module aims to learn the relations about neighbors across modalities. Finally, the proposed method can aggregate the neighborhood features from both intra modality and inter modality at the same time, thus performs better on image-text matching for considering much more semantic information. Extensive experimental results on MS COCO and Flickr30K show the superior performance of our proposed modal over the state-of-the-arts.