Multiple deep neural networks with multiple labels for cross-modal hashing retrieval

Abstract

Most deep hashing methods for cross-modal retrieval use semantic labels to judge simply whether a pair of data are similar or dissimilar. However, they do not make full use of the different labels between the two instances. In addition, they do not generate more discriminative hash codes that scatter around the class centre. To this end, we propose multiple deep neural networks with multiple labels for cross-modal hashing retrieval (MDMCH). MDMCH constructs a multiple deep hash learning framework that contains three-stream deep neural networks for images, texts, and labels. In terms of the objective function, the semantic weight and class centre similarity are calculated according to the multi-labels. Then, the semantic weight is embedded into three cross-entropy loss terms for image, text, and label networks to better preserve inter-modal and intra-modal similarities. Meanwhile, MDMCH fuses the deep features of semantic labels and texts to further improve the text-to-image retrieval accuracy. Additionally, the class centre similarity is applied to the image and text networks, which forces the similar instance pairs to scatter around the class centre. Experiments on three popular benchmark datasets demonstrate that the proposed method outperforms the state-of-the-art methods.