Multimodal Hashing Method Research Articles

Adaptive semi-paired query hashing for multi-modal retrieval

Multi-modal hashing has attracted enormous attention in large-scale multimedia retrieval, owing to its advantages of low storage cost and fast Hamming distance computation. Existing multi-modal hashing methods assume that all multi-modal data are well paired and then encode the paired multiple modalities into joint binary codes. However, it is not ensured that all data are fully paired in practical applications. In this paper, we present an adaptive semi-paired query hashing method, which facilitates learning the hash codes for semi-paired query samples. The proposed method performs projection learning and cross-modal reconstruction learning to maintain the semantic consistency between multi-modal data. Meanwhile, the semantic similarity structure and the complementary multi-modal information are preserved by hash codes to obtain a discriminative hash function. In the encoding stage, the missing modality features are completed via the learned cross-modal reconstruction matrices. In addition, the multimodal fusion weights are fine-tuned adaptively for the new query data to capture the modality difference. The extensive experiment results on three benchmark datasets show that our proposed algorithm outperforms state-of-the-art multi-modal hashing methods.

One for more: Structured Multi-Modal Hashing for multiple multimedia retrieval tasks

Hashing has been widely studied in support of efficient multimedia retrieval due to its high computation and storage efficiency. However, existing multimedia hashing models are designed only for specific multimedia retrieval tasks, e.g., cross-modal retrieval and composite multi-modal retrieval. To simultaneously support multiple multimedia retrieval tasks, multiple different hashing models should be equipped in a multimedia retrieval system. This paper proposes a Structured Multi-Modal Hashing (SMMH) method, which can learn only a hash model for multiple multimedia retrieval tasks. Specifically, we generate structured hash codes by independently projecting the modality data into the Hamming space, where each part of hash codes preserves the specific characteristic of the specific modality data, and they are collaborated to represent the whole multi-modal data. To bridge the heterogeneous gap between multiple modalities, a fusion multi-modal space is constructed based on structured hash codes with label and pair-wise semantic supervision. Besides, we learn hash functions for composite multi-modal retrieval by projecting heterogeneous modalities into the collaborative multi-modal features. Finally, we develop a discrete hash optimization strategy to efficiently solve binary hash codes. Extensive experiments show that our model outperforms the strong baseline methods on multiple multimedia retrieval tasks. Source codes are available at https://github.com/ChaoqunZheng/SMMH.

Partial Multi-Modal Hashing via Neighbor-Aware Completion Learning

Multi-modal hashing technology can support large-scale multimedia retrieval well, because of its fast query speed and low storage consumption. Although many multi-modal hashing methods have been developed in the literature, they still suffer from three important problems: 1) Most multi-modal hashing methods assume that multi-modal data are complete. This ideal assumption limits their application to practical retrieval scenarios, where the modality-missing is common. 2) Existing partial multi-modal hashing methods directly model incomplete multi-modal data for hash learning, which may result in partial multi-modal semantics in the learned hash codes. 3) Most of the methods are based on the shallow learning framework, which inevitably suffers from limited representation capability. To solve the above problems, we propose a flexible deep partial multi-modal hash learning framework, named Neighbor-aware Completion Hashing (NCH). Our framework jointly performs the cross-modal completion learning for incomplete multi-modal data and the multi-modal hash learning. It can not only support model training with incomplete multi-modal data but also handle incomplete multi-modal queries. Besides, we design a neighbor-aware completion learning module to capture neighbor semantics and generate distribution-consistent completed features. Finally, we conduct extensive experiments to evaluate our method on both fully-paired and partial multi-modal retrieval scenarios. The experimental results verify the superiority of our proposed method over the state-of-the-art baselines. The experimental codes of our method are released at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/FutureTwT/NCH.</uri>

Hadamard matrix-guided multi-modal hashing for multi-modal retrieval

Multi-modal hashing can encode heterogeneous multi-modal data into compact binary codes, which has been extensively studied to solve large-scale multi-modal retrieval. However, since pioneer methods do not exploit fully the potential discriminative information in category labels and the rich complementary information between multi-modal data, their retrieval performance is limited. To address this problem, we propose a novel multi-modal hashing method that performs subspace learning and target feature learning in an overall framework. On the one hand, the proposed method captures the complementary information between multi-modal data by adaptive projection learning. To enhance the feature representation ability, the multi-modal spaces are reconstructed via the collective matrix factorization. On the other hand, the target binary codes that are predefined by the Hadamard matrix are softened into the learnable target features, which can promote the inter-class separability and preserve the intra-class difference. The extensive experiment results conducted on three public datasets show that the proposed method outperforms state-of-the-art methods.

Online Enhanced Semantic Hashing: Towards Effective and Efficient Retrieval for Streaming Multi-Modal Data

With the vigorous development of multimedia equipments and applications, efficient retrieval of large-scale multi-modal data has become a trendy research topic. Thereinto, hashing has become a prevalent choice due to its retrieval efficiency and low storage cost. Although multi-modal hashing has drawn lots of attention in recent years, there still remain some problems. The first point is that existing methods are mainly designed in batch mode and not able to efficiently handle streaming multi-modal data. The second point is that all existing online multi-modal hashing methods fail to effectively handle unseen new classes which come continuously with streaming data chunks. In this paper, we propose a new model, termed Online enhAnced SemantIc haShing (OASIS). We design novel semantic-enhanced representation for data, which could help handle the new coming classes, and thereby construct the enhanced semantic objective function. An efficient and effective discrete online optimization algorithm is further proposed for OASIS. Extensive experiments show that our method can exceed the state-of-the-art models. For good reproducibility and benefiting the community, our code and data are already publicly available.

Adaptive multi-modal fusion hashing via Hadamard matrix

Hashing plays an important role in information retrieval, due to its low storage and high speed of processing. As an effective multi-modal representation learning method, multi-modal hashing has received particular attention. Most of the existing multi-modal hashing methods adopt the fixed weighting factors to fuse multiple modalities for any query data, which cannot capture the variation among different queries. Besides, there are too much hyper-parameters in their models while it is time-consuming and labor-intensive to determine the proper parameters. The limitations may significantly hinder their promotion in practical applications. In this paper, we propose a simple, yet effective method that is inspired by the Hadamard matrix. On the one hand, our proposed method that involves a very few hyper-parameters is flexible. On the other hand, the complementary information between multi-modal data and the semantic discrimination information are preserved well in the hash codes. Extensive experimental results on four benchmark datasets show that the proposed framework is effective and achieves superior performance compared to state-of-the-art methods.

Efficient Semi-Supervised Multimodal Hashing With Importance Differentiation Regression.

Multi-modal hashing learns compact binary hash codes by collaborating heterogeneous multi-modal features at both the model training and online retrieval stages to support large-scale multimedia retrieval. Previous multi-modal hashing methods mainly focus on supervised and unsupervised hashing. The performance of supervised hashing largely relies on the number of labeled data, which is practically expensive to obtain. Unsupervised hashing methods cannot effectively capture the semantic correlations of multi-modal data without any labels for supervision. In this paper, we propose an Efficient Semi-supervised Multi-modal Hashing with Importance Differentiation Regression (ESMH-IDR) model, which can alleviate the existing problems by learning from both labeled and unlabeled data. Specifically, in this paper, we develop an efficient semi-supervised multi-modal hash code learning module. It learns the hash codes for labeled data in an efficient asymmetric way, and simultaneously performs nonlinear regression using the same projection matrix as the labeled samples to preserve the intrinsic data structure of unlabeled data. Besides, different from existing methods, we propose an importance differentiation regression strategy to learn hash functions by specially considering the different importance of hash codes learned from the labeled and unlabeled samples. Finally, we develop an efficient discrete optimization method guaranteed with convergence to iteratively solve the hash optimization problem. Experiments on several public multimedia retrieval datasets demonstrate the superiority of our proposed method on both retrieval effectiveness and efficiency. Our source codes and testing datasets can be obtained at https://github.com/ChaoqunZheng/ESMH.

Semantic-Driven Interpretable Deep Multi-Modal Hashing for Large-Scale Multimedia Retrieval

Multi-modal hashing focuses on fusing different modalities and exploring the complementarity of heterogeneous multi-modal data for compact hash learning. However, existing multi-modal hashing methods still suffer from several problems, including: 1) Almost all existing methods generate unexplainable hash codes. They roughly assume that the contribution of each hash code bit to the retrieval results is the same, ignoring the discriminative information embedded in hash learning and semantic similarity in hash retrieval. Moreover, the length of hash code is empirically set, which will cause bit redundancy and affect retrieval accuracy. 2) Most existing methods exploit shallow models which fail to fully capture higher-level correlation of multi-modal data. 3) Most existing methods adopt online hashing strategy based on immutable direct projection, which generates query codes for new samples without considering the differences of semantic categories. In this paper, we propose a Semantic-driven Interpretable Deep Multi-modal Hashing (SIDMH) method to generate interpretable hash codes driven by semantic categories within a deep hashing architecture, which can solve all these three problems in an integrated model. The main contributions are: 1) A novel deep multi-modal hashing network is developed to progressively extract hidden representations of heterogeneous modality features and deeply exploit the complementarity of multi-modal data. 2) Learning interpretable hash codes, with discriminant information of different categories distinctively embedded into hash codes and their different impacts on hash retrieval intuitively explained. Besides, the code length depends on the number of categories in the dataset, which can reduce the bit redundancy and improve the retrieval accuracy. 3) The semantic-driven online hashing strategy encodes the significant branches and discards the negligible branches of each query sample according to the semantics contained in it, therefore it could capture different semantics in dynamic queries. Finally, we consider both the nearest neighbor similarity and semantic similarity of hash codes. Experiments on several public multimedia retrieval datasets validate the superiority of the proposed method.

Fast Discrete Collaborative Multi-Modal Hashing for Large-Scale Multimedia Retrieval

Many achievements have been made on learning to hash for uni-modal and cross-modal retrieval. However, it is still an unsolved problem that how to directly and efficiently learn discriminative discrete hash codes for the multimedia retrieval, where both query and database samples are represented with heterogeneous multi-modal features. With this motivation, we propose a Fast Discrete Collaborative Multi-modal Hashing (FDCMH) method in this paper. We first propose an efficient collaborative multi-modal mapping that first transforms heterogeneous multi-modal features into the unified factors to exploit the complementarity of multi-modal features and preserve the semantic correlations in multiple modalities with linear computation and space complexity. Such shared factors also bridge the heterogeneous modality gap and remove the inter-modality redundancy. Further, we develop an asymmetric hashing learning module to simultaneously correlate the learned hash codes with low-level data distribution and high-level semantics. In particular, this design could avoid the challenging symmetric semantic matrix factorization and $O(n^2)$ O ( n 2 ) memory cost ( $n$ n is the number of training samples). It can support both computation and memory efficient discrete hash optimization. Experiments on several public multimedia retrieval datasets demonstrate the superiority of the proposed approach compared with state-of-the-art hashing techniques, in terms of both model learning efficiency and retrieval accuracy.

Joint and individual matrix factorization hashing for large-scale cross-modal retrieval

Multimodal hashing methods have gained considerable attention in recent years due to their effectiveness and efficiency for cross-modal similarity searches. Existing multimodal hashing methods either learn unified hash codes for different modalities or learn individual hash codes for each modality and then explore cross-correlations between them. Generally, learning unified hash codes tends to preserve the shared properties of multimodal data and learning individual hash codes tends to preserve the specific properties of each modality. There remains a crucial bottleneck regarding how to learn hash codes that simultaneously preserve the shared properties and specific properties of multimodal data. Therefore, we present a joint and individual matrix factorization hashing (JIMFH) method, which not only learns unified hash codes for multimodal data to preserve their common properties but also learns individual hash codes for each modality to retain its specific properties. The proposed JIMFH learns unified hash codes by joint matrix factorization, which jointly factorizes all modalities into a shared latent semantic space. In addition, JIMFH learns individual hash codes by individual matrix factorization, which separately factorizes each modality into a modal-specific latent semantic space. Finally, unified hash codes and individual hash codes are combined to obtain the final hash codes. In this way, hash codes learned by JIMFH can preserve both the shared properties and specific properties of multimodal data, and therefore the retrieval performance is enhanced. Comprehensive experiments show that the proposed JIMFH performs much better than many state-of-the-art methods on cross-modal retrieval applications.

Enhanced Discrete Multi-Modal Hashing: More Constraints Yet Less Time to Learn

Due to the exponential growth of multimedia data, multi-modal hashing as a promising technique to make cross-view retrieval scalable is attracting more and more attention. However, most of the existing multi-modal hashing methods either divide the learning process unnaturally into two separate stages or treat the discrete optimization problem simplistically as a continuous one, which leads to suboptimal results. Recently, a few discrete multi-modal hashing methods that try to address such issues have emerged, but they still ignore several important discrete constraints (such as the balance and decorrelation of hash bits). In this paper, we overcome those limitations by proposing a novel method named &#x201C;Enhanced Discrete Multi-modal Hashing (EDMH)&#x201D; which learns binary codes and hashing functions simultaneously from the pairwise similarity matrix of data, under the aforementioned discrete constraints. Although the model of EDMH looks a lot more complex than the other models for multi-modal hashing, we are actually able to develop a fast iterative learning algorithm for it, since the subproblems of its optimization all have closed-form solutions after introducing a couple of auxiliary variables. Our experimental results on three real-world datasets have revealed the usefulness of those previously ignored discrete constraints and demonstrated that EDMH not only performs much better than state-of-the-art competitors according to several retrieval metrics but also runs much faster than most of them.

Flexible Multi-modal Hashing for Scalable Multimedia Retrieval

Multi-modal hashing methods could support efficient multimedia retrieval by combining multi-modal features for binary hash learning at the both offline training and online query stages. However, existing multi-modal methods cannot binarize the queries, when only one or part of modalities are provided. In this article, we propose a novel Flexible Multi-modal Hashing (FMH) method to address this problem. FMH learns multiple modality-specific hash codes and multi-modal collaborative hash codes simultaneously within a single model. The hash codes are flexibly generated according to the newly coming queries, which provide any one or combination of modality features. Besides, the hashing learning procedure is efficiently supervised by the pair-wise semantic matrix to enhance the discriminative capability. It could successfully avoid the challenging symmetric semantic matrix factorization and O ( n 2 ) storage cost of semantic matrix. Finally, we design a fast discrete optimization to learn hash codes directly with simple operations. Experiments validate the superiority of the proposed approach.

Online Adaptive Supervised Hashing for Large-Scale Cross-Modal Retrieval

In recent years, with the continuous growth of multimedia data on the Internet, multimodal hashing has attracted increasing attention for its efficiency in large-scale cross-modal retrieval. Typically, most existing multimodal hashing methods are batch-based methods that cannot deal with the growing streaming data. Online multimodal hashing adopts online learning strategy to learn hash models incrementally, which can process large-scale streaming data. However, existing supervised online multimodal hashing methods still suffer from several limitations: (1) most methods cannot update hash codes of old data when hash model changes, which will hinder the retrieval accuracy. (2) the discrete optimizations of most methods for learning binary hash codes are less efficient or less effective. To address the above limitations, an efficient supervised online multimodal hashing method termed Online Adaptive Supervised Hashing (OASH) is proposed in this paper. OASH regresses class labels and training data to binary hash codes to learn discrete hash codes and hash functions with an efficient online optimization scheme. It learns hash functions incrementally with newly arriving data and updates hash codes with the latest hash model. By adaptively updating hash functions and hash codes with new data rather than accessing to old data, it gains much performance enhancement and computation savings. Extensive experiments on three benchmark data sets validate the superiority of OASH over state-of-the-art methods in both accuracy and efficiency.

Interval Type-2 Fuzzy Logic for Semisupervised Multimodal Hashing.

Retrieving nearest neighbors across correlated data in multiple modalities, such as image-text pairs on Facebook and video-tag pairs on YouTube, has become a challenging task due to the huge amount of data. Multimodal hashing methods that embed data into binary codes can boost the retrieving speed and reduce storage requirements. Since unsupervised multimodal hashing methods are usually inferior to supervised ones and the supervised ones require too much manually labeled data, the proposed method in this paper utilizes partially available labels to design a semisupervised multimodal hashing method. The labels for unlabeled data are treated as an interval type-2 fuzzy set and estimated by the available labels. By defuzzifying the estimated labels using hard partitioning, a supervised multimodal hashing method is used to generate binary codes. Experiments show that the proposed semisupervised method with 50% labels can get a medium performance among the compared supervised ones and achieve approximate performance to the best supervised method with 90% labels. With only 10% labels, the proposed method can still compete with the worst compared supervised one. Furthermore, the proposed label estimation method has been experimentally proven to be more feasible for a multilabeled MIRFlickr data set in a hash lookup task.

Multimedia retrieval by deep hashing with multilevel similarity learning

Deep multimodal hashing has received increasing research attention in recent years due to its superior performance for large-scale multimedia retrieval. However, limited e orts have been made to explore the complex multilevel semantic structure for deep multimodal hashing. In this paper, we propose a novel deep multimodal hashing method, termed as Deep Hashing with Multilevel Similarity Learning (DHMSL), for learning compact and discriminative hash codes, which explores multilevel semantic similarity correlations of multimedia data. In DHMSL, multilevel similarity correlation is explored to learn the unified binary hash codes by exploiting the local structure and semantic label information simultaneously. Meanwhile, the bit balance and quantization constraints are taken into account to further make the unified hash codes compact. With the unified binary codes learned, two deep neural networks are jointly trained to simultaneously learn feature representations and two sets of nonlinear hash functions. Specifically, the well-designed loss functions are introduced to minimize the prediction errors of the feature representations as well as the errors between the unified binary codes and outputs of the networks. Extensive experiments on two widely-used multimodal datasets demonstrate that the proposed method can achieve the state-of-the-art performance for both image-query-text and text-query-image tasks.

Semi-paired and semi-supervised multimodal hashing via cross-modality label propagation

Due to the fast query speed and low storage cost, multimodal hashing methods have been attracting increasing attention in large-scale cross-media retrieval tasks. Most existing multimodal hashing methods can only handle fully-paired settings, where all data samples with different modalities are well paired. However, in practical applications, such fully-paired multimodal data may not be available. To this end, semi-paired multimodal hashing methods have been proposed by exploiting correlations between unpaired samples. Nevertheless, currently existing semi-paired hashing methods are unsupervised methods. When little supervised information is available, these methods cannot utilize supervised information to enhance the retrieval performance. To effectively utilize the limited supervised information, this paper proposed a novel hashing framework, named semi-paired and semi-supervised multimodal hashing (SSMH), to deal with the scenario where partial pairwise correspondences and labels are provided in advance for cross-media retrieval task. The proposed SSMH propagates the semantic labels from labeled multimodal samples to unlabeled multimodal samples, so that the label information of the entire multimodal training set is available. Then, most existing similarity graph based supervised multimodal hashing methods can be used to learn hashing codes. Therefore, the proposed framework can fully utilize the limited label information and pairwise correspondences to keep the semantic similarity for hashing codes. Thorough experiments on standard datasets show the superior performance of the proposed framework.

Robust and Flexible Discrete Hashing for Cross-Modal Similarity Search

Multimodal hashing approaches have gained great success on large-scale cross-modal similarity search applications, due to their appealing computation and storage efficiency. However, it is still a challenge work to design binary codes to represent the original features with good performance in an unsupervised manner. We argue that there are some limitations that need to be further considered for unsupervised multimodal hashing: 1) most existing methods drop the discrete constraints to simplify the optimization, which will cause large quantization error; 2) many methods are sensitive to outliers and noises since they use $\ell _{2}$ -norm in their objective functions which can amplify the errors; and 3) the weight of each modality, which greatly influences the retrieval performance, is manually or empirically determined and may not fully fit the specific training set. The above limitations may significantly degrade the retrieval accuracy of unsupervised multimodal hashing methods. To address these problems, in this paper, a novel hashing model is proposed to efficiently learn robust discrete binary codes, which is referred as Robust and Flexible Discrete Hashing (RFDH). In the proposed RFDH model, binary codes are directly learned based on discrete matrix decomposition, so that the large quantization error caused by relaxation is avoided. Moreover, the $\ell _{2,1}$ -norm is used in the objective function to improve the robustness, such that the learned model is not sensitive to data outliers and noises. In addition, the weight of each modality is adaptively adjusted according to training data. Hence the important modality will get large weights during the hash learning procedure. Owing to above merits of RFDH, it can generate more effective hash codes. Besides, we introduce two kinds of hash function learning methods to project unseen instances into hash codes. Extensive experiments on several well-known large databases demonstrate superior performance of the proposed hash model over most state-of-the-art unsupervised multimodal hashing methods.

Label Consistent Matrix Factorization Hashing for Large-Scale Cross-Modal Similarity Search.

Multimodal hashing has attracted much interest for cross-modal similarity search on large-scale multimedia data sets because of its efficiency and effectiveness. Recently, supervised multimodal hashing, which tries to preserve the semantic information obtained from the labels of training data, has received considerable attention for its higher search accuracy compared with unsupervised multimodal hashing. Although these algorithms are promising, they are mainly designed to preserve pairwise similarities. When semantic labels of training data are given, the algorithms often transform the labels into pairwise similarities, which gives rise to the following problems: (1) constructing pairwise similarity matrix requires enormous storage space and a large amount of calculation, making these methods unscalable to large-scale data sets; (2) transforming labels into pairwise similarities loses the category information of the training data. Therefore, these methods do not enable the hash codes to preserve the discriminative information reflected by labels and, hence, the retrieval accuracies of these methods are affected. To address these challenges, this paper introduces a simple yet effective supervised multimodal hashing method, called label consistent matrix factorization hashing (LCMFH), which focuses on directly utilizing semantic labels to guide the hashing learning procedure. Considering that relevant data from different modalities have semantic correlations, LCMFH transforms heterogeneous data into latent semantic spaces in which multimodal data from the same category share the same representation. Therefore, hash codes quantified by the obtained representations are consistent with the semantic labels of the original data and, thus, can have more discriminative power for cross-modal similarity search tasks. Thorough experiments on standard databases show that the proposed algorithm outperforms several state-of-the-art methods.

Semantic Neighbor Graph Hashing for Multimodal Retrieval.

Hashing methods have been widely used for approximate nearest neighbor search in recent years due to its computational and storage effectiveness. Most existing multimodal hashing methods try to preserve the similarity relationship based on either metric distances or semantic labels in a procrustean way, while ignoring the intra-class and inter-class variations inherent in the metric space. In this paper, we propose a novel multimodal hashing method, termed as semantic neighbor graph hashing (SNGH), which aims to preserve the fine-grained similarity metric based on the semantic graph that is constructed by jointly pursuing the semantic supervision and the local neighborhood structure. Specifically, the semantic graph is constructed to capture the local similarity structure for the image modality and the text modality, respectively. Furthermore, we define a function based on the local similarity in particular to adaptively calculate multi-level similarities by encoding the intra-class and inter-class variations. After obtaining the unified hash codes, the logistic regression with kernel trick is employed to learn view-specific hash functions independently for each modality. Extensive experiments are conducted on four widely used multimodal data sets. The experimental results demonstrate the superiority of the proposed SNGH method compared with the state-of-the-art multimodal hashing methods.

Diversity Regularized Latent Semantic Match for Hashing

Hashing based approximate nearest neighbors (ANN) search has drawn considerable attraction owing to its low-memory storage and hardware-level logical computing which is doomed to be greatly applicable to quantities of large-scale and practical scenarios, such as information retrieval, computer vision and natural language processing. However, most existing hashing methods concentrate either on images only or on pairwise image-texts (labels, short documents) and rarely utilize more common sentences. In this paper, we propose D iversity R egularized L atent S emantic M atch for H ashing (DRLSMH), a new multimodal hashing method that projects images and sentences into a shared latent semantic space with label-supervised semantic constraints to proceed on multimodal retrieval. Notably, soft orthogonality is induced as a novel regularizer to preserve diverse hashing functions for compact and accurate representations; what's more, this kind of regularization also benefits the derivations of closed-form solutions with some proper relaxations under iterative optimization framework. Extensive experiments on two public datasets demonstrate the advantages of our method over some state-of-the-art baselines under cross-modal retrieval both on image-query-image, image-query-text and text-query-image tasks.