Cross-modal Hashing Research Articles

Deep Supervised Cross-modal Hashing for Ship Image Retrieval

The retrieval of multimodal ship images obtained by remote sensing satellites is an important content of remote sensing data analysis, which is of great significance to improve the ability of marine monitoring. In this paper, We propose a novel cross-modal ship image retrieval method, called Deep Supervised Cross-modal Hashing(DSCMH). It consists of a feature learning part and a hash learning part used for feature extraction and hash code generation separately, both two parts have modality-invariant constraints to keep the cross-modal invariability, and the label information is also brought to supervise the above process. Furthermore, we design a class attention module based on the cross-modal class center to strengthen class discrimination. The experiment results show that the proposed method can effectively improve the cross-modal retrieval accuracy of ship images and is better than several state-of-the-art methods.

Unsupervised Deep Relative Neighbor Relationship Preserving Cross-Modal Hashing

The image-text cross-modal retrieval task, which aims to retrieve the relevant image from text and vice versa, is now attracting widespread attention. To quickly respond to the large-scale task, we propose an Unsupervised Deep Relative Neighbor Relationship Preserving Cross-Modal Hashing (DRNPH) to achieve cross-modal retrieval in the common Hamming space, which has the advantages of storage and efficiency. To fulfill the nearest neighbor search in the Hamming space, we demand to reconstruct both the original intra- and inter-modal neighbor matrix according to the binary feature vectors. Thus, we can compute the neighbor relationship among different modal samples directly based on the Hamming distances. Furthermore, the cross-modal pair-wise similarity preserving constraint requires the similar sample pair have an identical Hamming distance to the anchor. Therefore, the similar sample pairs own the same binary code, and they have minimal Hamming distances. Unfortunately, the pair-wise similarity preserving constraint may lead to an imbalanced code problem. Therefore, we propose the cross-modal triplet relative similarity preserving constraint, which demands the Hamming distances of similar pairs should be less than those of dissimilar pairs to distinguish the samples’ ranking orders in the retrieval results. Moreover, a large similarity marginal can boost the algorithm’s noise robustness. We conduct the cross-modal retrieval comparative experiments and ablation study on two public datasets, MIRFlickr and NUS-WIDE, respectively. The experimental results show that DRNPH outperforms the state-of-the-art approaches in various image-text retrieval scenarios, and all three proposed constraints are necessary and effective for boosting cross-modal retrieval performance.

Discrete Fusion Adversarial Hashing for cross-modal retrieval

Deep cross-modal hashing enables a flexible and efficient way for large-scale cross-modal retrieval. Existing cross-modal retrieval methods based on deep hashing aim to learn the unified hashing representation for different modalities with the supervision of pair-wise correlation, and then encode the out-of-samples via modality-specific hashing network. However, the semantic gap and distribution shift were not considered enough, and the hashing codes cannot be unified as expected under different modalities. At the same time, hashing is still a discrete problem that has not been solved well in the deep neural network. Therefore, we propose the Discrete Fusion Adversarial Hashing (DFAH) network for cross-modal retrieval to address these issues. In DFAH, the Modality-Specific Feature Extractor is designed to capture image and text features with pair-wise supervision. Especially, the Fusion Learner is proposed to learn the unified hash code, which enhances the correlation of heterogeneous modalities via the embedding strategy. Meanwhile, the Modality Discriminator is designed to adapt to the distribution shift cooperating with the Modality-Specific Feature Extractor in an adversarial way. In addition, we design an efficient discrete optimization strategy to avoid the relaxing quantization errors in the deep neural framework. Finally, the experiment results and analysis on several popular datasets also show that DFAH outperforms the state-of-the-art methods for cross-modal retrieval.

Multiple deep neural networks with multiple labels for cross-modal hashing retrieval

Most deep hashing methods for cross-modal retrieval use semantic labels to judge simply whether a pair of data are similar or dissimilar. However, they do not make full use of the different labels between the two instances. In addition, they do not generate more discriminative hash codes that scatter around the class centre. To this end, we propose multiple deep neural networks with multiple labels for cross-modal hashing retrieval (MDMCH). MDMCH constructs a multiple deep hash learning framework that contains three-stream deep neural networks for images, texts, and labels. In terms of the objective function, the semantic weight and class centre similarity are calculated according to the multi-labels. Then, the semantic weight is embedded into three cross-entropy loss terms for image, text, and label networks to better preserve inter-modal and intra-modal similarities. Meanwhile, MDMCH fuses the deep features of semantic labels and texts to further improve the text-to-image retrieval accuracy. Additionally, the class centre similarity is applied to the image and text networks, which forces the similar instance pairs to scatter around the class centre. Experiments on three popular benchmark datasets demonstrate that the proposed method outperforms the state-of-the-art methods.

Bi-CMR: Bidirectional Reinforcement Guided Hashing for Effective Cross-Modal Retrieval

Cross-modal hashing has attracted considerable attention for large-scale multimodal data. Recent supervised cross-modal hashing methods using multi-label networks utilize the semantics of multi-labels to enhance retrieval accuracy, where label hash codes are learned independently. However, all these methods assume that label annotations reliably reflect the relevance between their corresponding instances, which is not true in real applications. In this paper, we propose a novel framework called Bidirectional Reinforcement Guided Hashing for Effective Cross-Modal Retrieval (Bi-CMR), which exploits a bidirectional learning to relieve the negative impact of this assumption. Specifically, in the forward learning procedure, we highlight the representative labels and learn the reinforced multi-label hash codes by intra-modal semantic information, and further adjust similarity matrix. In the backward learning procedure, the reinforced multi-label hash codes and adjusted similarity matrix are used to guide the matching of instances. We construct two datasets with explicit relevance labels that reflect the semantic relevance of instance pairs based on two benchmark datasets. The Bi-CMR is evaluated by conducting extensive experiments over these two datasets. Experimental results prove the superiority of Bi-CMR over four state-of-the-art methods in terms of effectiveness.

Scalable semantic-enhanced supervised hashing for cross-modal retrieval

Cross-modal retrieval is a classic scenario that can describe the same semantics from multiple angles and provide us with more abundant and diversified information ; thus, it has abundant applications in information retrieval, data mining, and machine learning. Generally, supervised information can enhance the ability to learn hash features, and obtain better retrieval accuracy than unsupervised methods. However, how to effectively solve the discrete binary constraint problem, and how to retain more semantic information in the process of hash learning, which constitute important parts of cross-modal retrieval, still have some challenges. To mitigate these problems, in this work, we propose a novel cross-modal hashing approach, called, scalable pairwise embedding constraint hashing (SPECH). The SPECH approach employs the loss of likelihood similarity technique using pairwise sample data to measure the semantic similarity of heterogeneous modal samples. In light of this, the SPECH approach can maximize the utilization of available data from heterogeneous modal samples for training models. In addition, we further enhance the discriminative capability of hash codes and reduce the intraclass differences in heterogeneous modal samples, by embedding the more discriminative semantic label attribute space regression into the to-be-learned process of hash codes. Comprehensive experiments on three benchmark datasets show that the proposed SPECH approach achieves significantly better retrieval accuracy and outperforms several state-of-the-art approaches on cross-modal retrieval applications.

Asymmetric cross–modal hashing with high–level semantic similarity

Cross-modal hashing aims at using modality content to retrieve semantically relevant objects of different modalities, so cross-modal retrieval has attracted much attention. To effectively exploit the discriminative label information and retain more semantic information in the process of hash learning, we propose a novel cross-modal hashing method, named high-level semantic similarity analysis hashing (HSSAH) for cross-modal retrieval. To reduce time complexity and enhance discriminant ability in hash codes, HSSAH constructs an asymmetric high-level semantic similarity learning framework to replace the binary semantic similarity matrix. Moreover, the developed HSSAH is a two-stage approach, and a semantic-enhanced scheme is proposed in the second stage, which fully leverages the label information to gain more powerful hash functions. We conducted comprehensive experiments on three benchmark datasets to evaluate the performance of HSSAH. Experimental results show that HSSAH can achieve significantly better retrieval precision and outperforms several state-of-the-art approaches.

Enhancing Cross-Modal Hash Retrieval with Multiple Similarity Matrices

In order to further improve the performance of cross-modal retrieval, a cross-modal hash retrieval method integrating multi-level similarity information is proposed. First, self-attention method is used to enhance the text features, and a new fusion feature is constructed based on the original features and hash features of different modalities. Then, based on these three features, three auxiliary similarity matrices are constructed, and the fourth auxiliary similarity matrix is constructed by a weighted combination method. Finally, these four different matrices are used to calculate the loss functions between different similarity matrices and between different modalities. Since the four matrices include different feature forms and different matrix construction methods, they can better express similarity information of different modalities and improve the retrieval performance. The experiments are conducted on three benchmark datasets of Wikipedia, MIRFlickr and NUS-WIDE. The results show that the mAP values at different code bits of proposed method is better than that of many state-of-the-art methods, which verifies the effectiveness and robustness of our method.

Semi-supervised cross-modal hashing with multi-view graph representation

Recently, significant progress has been made in graph-based hashing methods for the purpose of learning hash codes that can preserve semantic similarity. Many approaches have been formulated for supervised learning problems that require labels. However, large-scale labeled datasets are expensive to obtain, especially when the data are multimodal, thus imposing a restriction on the usage of such algorithms. In this study, a novel multi-view graph cross-modal hashing (MGCH) framework is proposed to generate hash codes in a semi-supervised manner using the outputs of multi-view graphs processed by a graph-reasoning module. In contrast to conventional graph-based hashing methods, MGCH adopts multi-view graphs as the only learning assistance to connect labeled and unlabeled data in the process of pursuing binary embeddings. Multiview graphs that filter the features of multidirectional data in multiple anchor sets are beneficial in refining features. As the core component of our MGCH, an intuitive graph reasoning network consisting of two graph convolutional layers and one graph attention layer is employed to simultaneously convolve anchor graphs and asymmetric graphs with input data. Comprehensive cross-modal hashing evaluations on the Wiki, MIRFlickr-25K, NUS-WIDE, and MSCOCO datasets demonstrate the superiority of MGCH over the latest methods for limited labeled data.

Gaussian similarity preserving for cross-modal hashing

Cross-modal hashing (CMH) has gained considerable attention due to its low storage and fast query speed. Most existing CMH research pursues the binary codes of each modality by preserving a similarity matrix which is computationally expensive. In this paper, we propose a Gaussian similarity preserving method for cross-modal hashing. By using the semantic transformation, our model can avoid computing the similarity matrix explicitly. We further employ the sequential learning approach to reduce the quantization error. Experimental results on three benchmark datasets clearly show that our proposed method can outperform the state-of-the-art methods.

A cover selection-based reversible data hiding method by learning cross-modal hashing

A cover selection-based reversible data hiding method by learning cross-modal hashing

Object-Level Visual-Text Correlation Graph Hashing for Unsupervised Cross-Modal Retrieval.

The core of cross-modal hashing methods is to map high dimensional features into binary hash codes, which can then efficiently utilize the Hamming distance metric to enhance retrieval efficiency. Recent development emphasizes the advantages of the unsupervised cross-modal hashing technique, since it only relies on relevant information of the paired data, making it more applicable to real-world applications. However, two problems, that is intro-modality correlation and inter-modality correlation, still have not been fully considered. Intra-modality correlation describes the complex overall concept of a single modality and provides semantic relevance for retrieval tasks, while inter-modality correction refers to the relationship between different modalities. From our observation and hypothesis, the dependency relationship within the modality and between different modalities can be constructed at the object level, which can further improve cross-modal hashing retrieval accuracy. To this end, we propose a Visual-textful Correlation Graph Hashing (OVCGH) approach to mine the fine-grained object-level similarity in cross-modal data while suppressing noise interference. Specifically, a novel intra-modality correlation graph is designed to learn graph-level representations of different modalities, obtaining the dependency relationship of the image region to image region and the tag to tag in an unsupervised manner. Then, we design a visual-text dependency building module that can capture correlation semantic information between different modalities by modeling the dependency relationship between image object region and text tag. Extensive experiments on two widely used datasets verify the effectiveness of our proposed approach.

An efficient dual semantic preserving hashing for cross-modal retrieval

Hashing methods have recently received widespread attention due to their flexibility and effectiveness for cross-modal retrieval tasks. However, most existing cross-modal hashing methods have some challenging problems, in particular, effective exploitation of semantic information and learning discriminative hash codes. To address these challenges, we propose an efficient Dual Semantic Preserving Hashing (DSPH) method, which first leverages matrix factorization to obtain low-level latent semantic representations of different modalities and remove redundant information. To enhance the discriminative capability of hash codes, we preserve the high-level pairwise semantics and the learned low-level latent semantics into the unified hash codes. Finally, DSPH adopts discrete optimization strategy to learn the hash codes directly. Experimental results on three benchmark datasets demonstrate that the proposed DSPH method outperforms many state-of-the-art cross-modal hashing methods in terms of retrieval accuracy, especially when dealing with short hash code.

MS[formula omitted]GAH: Multi-label semantic supervised graph attention hashing for robust cross-modal retrieval

Due to the strong nonlinear representation capabilities of deep neural networks and the low storage and high efficiency characteristics of hash learning, deep cross-modal hashing has been propelled to the forefront of academics. How to preferably bridge semantic relevance to further bridge the semantic modality gap is the vital bottleneck to improve model performance. Confronting samples with rich semantics, how to comprehensively explore the hidden correlations and establish more precise modality relationships is the primary issue to be solved. In this work, we propose a novel deep hashing method called Multi-Label Semantic Supervised Graph Attention Hashing (MS2GAH), which is an end-to-end framework that integrates graph attention networks (GATs). It constructs graph features through the adjacency of nodes and assigns different weights to adjacent edges to enhance the robustness of the model. Simultaneously, multi-label annotations are utilized to bridge the semantic relevance between modalities in a more fine-grained manner. To make preferable use of rich semantic information, an end-to-end label encoder is designed to mine high-level semantics from multi-label annotations to guide the feature extraction process of specific-modality networks, thereby further narrowing the modality gap. Finally, extensive experiments have been conducted on four datasets, and the results show that MS2GAH is superior to other baselines and one step forward.

Semantic Constraints Matrix Factorization Hashing for cross-modal retrieval

Cross-modal hashing methods have attracted considerable attention due to their low memory usage and high query speed in large-scale cross-modal retrieval. During the encoding process, there still remains two crucial bottlenecks: how to equip hash codes with cross-modal semantic information, and how to rapidly obtain hash codes. In this paper, we propose Semantic Constraints Matrix Factorization Hashing (SCMFH) which simultaneously considers modality-specific and cross-modal semantic information for hash codes with closed solution. Specifically, the original representation of each modality is factorized into modality-specific semantic representations, and then the cross-modal semantic similarity matrices is used to constrain correlation of modality-specific representation. Finally, all the modality-specific representations are regressed to the common hash codes, and the fast hash codes with a closed solution is obtained. Extensive experimental have been carried out on public datasets. The results show that the method outperforms many current cross-modal hashing methods in terms of mean average precision (mAP), up to 2.7%.

Deep fused two-step cross-modal hashing with multiple semantic supervision

Deep fused two-step cross-modal hashing with multiple semantic supervision

Cross‐modal retrieval based on deep regularized hashing constraints

Cross-modal retrieval has attracted great attention due to the increasing demand for tremendous amounts of multimodal data in recent years. These retrievals could either be text-to-image or image-to-text. To address the problem of inappropriate information included between images and texts, we propose two cross-modal recovery techniques established on a dual-branch neural network defined on a common subspace and the hashing learning method. First, a cross-modal recovery technique established on a multilabel information deep ranking model (MIDRM) is provided. In this method, we introduce a triplet-loss function into the dual-branch neural network model. This function takes advantage of the semantic information of the bimodal components, focusing on not only the similarities between similar images and text features but also the distances between dissimilar images and texts. Second, we establish a new cross-modal hashing technique said to be the deep regularized hashing constraint (DRHC). In this method, the regularized function is used to replace the binary constraint, and the discrete value is constrained to a certain numerical range so that the network can achieve end-to-end training. Overall, the time complexity is greatly improved, and the occupied storage space is also greatly reduced. Different experiments on our proposed MIDRM and DRHC models demonstrate their superior performance to those of the state-of-the-art methods on two widely used data sets. The experimental results show that our approach also increases the mean average precision of cross-modal recovery.

Category supervised cross-modal hashing retrieval for chest X-ray and radiology reports

With the wide application of radiography, massive data of chest X-ray and the associated radiology reports have been accumulated. Cross-modal retrieval between the chest X-ray and radiology reports is useful for the level of practicing radiologists and substantial other medical settings. However, existing cross-modal retrieval methods, which are mainly designed for the natural images, are not suitable for the chest X-ray images and radiology reports. In this paper, we propose a category supervised cross-modal hashing retrieval between chest X-ray and radiology reports, which learns the cross-modal similarity using a category supervised hashing network and union hashing network. Specifically, we design a category hashing network to learn hash code for each category, then use the learned category hash code as supervised information to guide the learning of images modality hash, texts modality hash. On the other hand, we propose the union hashing network to learn the correlation between two different modalities. Comprehensive experiments have been done on the public dataset MIMIC-CXR and the results show that the proposed method is on average 6.62% better than the traditional shallow method, achieved an increase of 0.57% compared to the deep cross-modal hashing (DCMH) in term of mAP. The ablation study is also implemented and the results demonstrates effectiveness of the proposed method.

Deep Multi-Semantic Fusion-Based Cross-Modal Hashing

Due to the low costs of its storage and search, the cross-modal retrieval hashing method has received much research interest in the big data era. Due to the application of deep learning, the cross-modal representation capabilities have risen markedly. However, the existing deep hashing methods cannot consider multi-label semantic learning and cross-modal similarity learning simultaneously. That means potential semantic correlations among multimedia data are not fully excavated from multi-category labels, which also affects the original similarity preserving of cross-modal hash codes. To this end, this paper proposes deep multi-semantic fusion-based cross-modal hashing (DMSFH), which uses two deep neural networks to extract cross-modal features, and uses a multi-label semantic fusion method to improve cross-modal consistent semantic discrimination learning. Moreover, a graph regularization method is combined with inter-modal and intra-modal pairwise loss to preserve the nearest neighbor relationship between data in Hamming subspace. Thus, DMSFH not only retains semantic similarity between multi-modal data, but integrates multi-label information into modal learning as well. Extensive experimental results on two commonly used benchmark datasets show that our DMSFH is competitive with the state-of-the-art methods.

Semantic-guided autoencoder adversarial hashing for large-scale cross-modal retrieval

With the vigorous development of mobile Internet technology and the popularization of smart devices, while the amount of multimedia data has exploded, its forms have become more and more diversified. People’s demand for information is no longer satisfied with single-modal data retrieval, and cross-modal retrieval has become a research hotspot in recent years. Due to the strong feature learning ability of deep learning, cross-modal deep hashing has been extensively studied. However, the similarity of different modalities is difficult to measure directly because of the different distribution and representation of cross-modal. Therefore, it is urgent to eliminate the modal gap and improve retrieval accuracy. Some previous research work has introduced GANs in cross-modal hashing to reduce semantic differences between different modalities. However, most of the existing GAN-based cross-modal hashing methods have some issues such as network training is unstable and gradient disappears, which affect the elimination of modal differences. To solve this issue, this paper proposed a novel Semantic-guided Autoencoder Adversarial Hashing method for cross-modal retrieval (SAAH). First of all, two kinds of adversarial autoencoder networks, under the guidance of semantic multi-labels, maximize the semantic relevance of instances and maintain the immutability of cross-modal. Secondly, under the supervision of semantics, the adversarial module guides the feature learning process and maintains the modality relations. In addition, to maintain the inter-modal correlation of all similar pairs, this paper use two types of loss functions to maintain the similarity. To verify the effectiveness of our proposed method, sufficient experiments were conducted on three widely used cross-modal datasets (MIRFLICKR, NUS-WIDE and MS COCO), and compared with several representatives advanced cross-modal retrieval methods, SAAH achieved leading retrieval performance.