Deep Semantic Correlation with Adversarial Learning for Cross-Modal Retrieval

Abstract

Correlation learning usually maps heterogeneous data into a common subspace to achieve cross-modal retrieval. Thanks to the success of deep learning in recent years, the performance of cross-modal retrieval has made a great improvement. However, how to bridge the modality gap is still the key problem. In this paper, we propose a deep semantic correlation learning method with generative adversarial network to deal with cross-modal data annotated by multi-labels. With adversarial learning, the generative network tries to produce the common semantic representations respect to image and text modalities, while discriminative model tries to point out the differences between them. Besides that, we propose a classification loss applied to one or multiple categories for semantic subspace learning to promote cross-modal retrieval. The adversarial network and the classification network are jointly optimized. Experiments verify the effectiveness of our proposed model on two widely used datasets.