Semantic-enhanced discrete matrix factorization hashing for heterogeneous modal matching

Abstract

The simultaneous hash representation of heterogeneous modalities has shown its excellent performance in the multi-modal learning community. However, the binary discrete constraint of hash brings a great challenge to the common hash representation of heterogeneous modal data. To resolve the issue, this paper proposes a novel discrete hash method, referred to as Semantic enhanced Discrete Matrix Factorization Hashing (SDMFH). SDMFH directly extracts the common discrete hash representation of all modalities from the reconstructed semantic intermodal similarity graph, which makes the hash codes more discriminative. Meanwhile, the semantic labels are regressed to the extracted discrete hash book, so as to further strengthen the discriminating ability of the learned discrete hash book. Moreover, a linear embedding from the kernel space of the original data to the Hamming space is explored to generate the common hash codes of each modality to ensure the consistency of the hash codes of heterogeneous modalities. More importantly, we develop an efficient discrete iterative algorithm based on Stiefel manifold that can directly learn the discrete hash book in a closed form, thus both avoiding quantization loss caused by discrete relaxation and reducing computational complexity. Experimental results on three benchmark data sets show that SDMFH performs better than several state-of-the-art methods for heterogeneous modal matching tasks.