Multi-task framework based on feature separation and reconstruction for cross-modal retrieval

Abstract

Cross-modal retrieval has become a hot research topic in both computer vision and natural language processing areas. Learning intermediate common space for features of different modalities has become one of mainstream methods. In this paper, we propose a novel multi-task framework based on feature separation and reconstruction (mFSR) for cross-modal retrieval based on common space learning methods, which introduces feature separation module to deal with information asymmetry between different modalities, and introduces image and text reconstruction module to improve the quality of feature separation module. Extensive experiments on MS-COCO and Flickr30K datasets demonstrate that feature separation and specific information reconstruction can significantly improve the baseline performance of cross-modal image-caption retrieval.