Multi-Scale Correlation for Sequential Cross-modal Hashing Learning

Abstract

Cross-modal hashing aims to learn hash functions, which map heterogeneous multimedia data into common Hamming space for fast and flexible cross-modal retrieval. Recently, several cross-modal hashing methods learn the hash functions by mining the correlation among multimedia data. However, they ignore two properties of cross-modal data: 1) The features of different scale in single modality consist different information, such as texture, object and scene feature in the image, which can provide multi-scale information on retrieval task. 2) The correlation among the features of different modalities and scales can provide multi-scale relationship for better cross-modal hashing learning. In this paper, we propose Multi-scale Correlation Sequential Cross-modal Hashing Learning (MCSCH) approach. The main contributions of the MCSCH can be summarized as follows: 1) We propose a multi-scale feature guided sequential hashing learning method which sequentially generates the hash code guided by different scale features through a RNN based network. The multi-scale feature guided sequential hashing learning method utilizes the scale information, which enhances the diversity of the hash codes and reduces the error caused by extreme situation in specifc features. 2) We propose a multi-scale correlation mining strategy during the multi-scale feature guided sequential hashing learning, which can simultaneously mine the correlation among the features of different modalities and scales. Through this strategy, we can mine any pair of scale features in different modalities and obtain abundant scale correlation for better cross-modal retrieval. Experiments on two widely-used datasets demonstrate the effectiveness of our proposed MCSCH approach.