One for more: Structured Multi-Modal Hashing for multiple multimedia retrieval tasks

Abstract

Hashing has been widely studied in support of efficient multimedia retrieval due to its high computation and storage efficiency. However, existing multimedia hashing models are designed only for specific multimedia retrieval tasks, e.g., cross-modal retrieval and composite multi-modal retrieval. To simultaneously support multiple multimedia retrieval tasks, multiple different hashing models should be equipped in a multimedia retrieval system. This paper proposes a Structured Multi-Modal Hashing (SMMH) method, which can learn only a hash model for multiple multimedia retrieval tasks. Specifically, we generate structured hash codes by independently projecting the modality data into the Hamming space, where each part of hash codes preserves the specific characteristic of the specific modality data, and they are collaborated to represent the whole multi-modal data. To bridge the heterogeneous gap between multiple modalities, a fusion multi-modal space is constructed based on structured hash codes with label and pair-wise semantic supervision. Besides, we learn hash functions for composite multi-modal retrieval by projecting heterogeneous modalities into the collaborative multi-modal features. Finally, we develop a discrete hash optimization strategy to efficiently solve binary hash codes. Extensive experiments show that our model outperforms the strong baseline methods on multiple multimedia retrieval tasks. Source codes are available at https://github.com/ChaoqunZheng/SMMH.