Semantic-Consistent Deep Quantization for Cross-modal Retrieval

Abstract

With making up for the deficiency of the constraint representation capability of hashing codes for high-dimensional data, the quantization method has been found to generally perform better in cross-modal similarity retrieval research. However, in current quantization approaches, the codebook, as the most critical basis for quantization, is still in a passive status and detached from the learning framework. To improve the initiative of codebook, we propose a semantic-consistent deep quantization (SCDQ), which is the first scheme to integrate quantization into deep network learning in an end-to-end fashion. Specifically, two classifiers following the deep representation learning networks are formulated to produce the class-wise abstract patterns with the help of label alignment. Meanwhile, our approach learns a collaborative codebook for both modalities, which embeds bimodality semantic consistent information in codewords and bridges the relationship between the patterns in classifiers and codewords in codebook. By designing a novel algorithm architecture and codebook update strategy, SCDQ enables effective and efficient cross-modal retrieval in an asymmetric way. Extensive experiments on two benchmark datasets demonstrate that SCDQ yields optimal cross-modal retrieval performance and outperforms several state of-the-art cross-modal retrieval methods.