Deep Hashing Research Articles

Triplet-constrained deep hashing for chest X-ray image retrieval in COVID-19 assessment

Radiology images of the chest, such as computer tomography scans and X-rays, have been prominently used in computer-aided COVID-19 analysis. Learning-based radiology image retrieval has attracted increasing attention recently, which generally involves image feature extraction and finding matches in extensive image databases based on query images. Many deep hashing methods have been developed for chest radiology image search due to the high efficiency of retrieval using hash codes. However, they often overlook the complex triple associations between images; that is, images belonging to the same category tend to share similar characteristics and vice versa. To this end, we develop a triplet-constrained deep hashing (TCDH) framework for chest radiology image retrieval to facilitate automated analysis of COVID-19. The TCDH consists of two phases, including (a) feature extraction and (b) image retrieval. For feature extraction, we have introduced a triplet constraint and an image reconstruction task to enhance discriminative ability of learned features, and these features are then converted into binary hash codes to capture semantic information. Specifically, the triplet constraint is designed to pull closer samples within the same category and push apart samples from different categories. Additionally, an auxiliary image reconstruction task is employed during feature extraction to help effectively capture anatomical structures of images. For image retrieval, we utilize learned hash codes to conduct searches for medical images. Extensive experiments on 30,386 chest X-ray images demonstrate the superiority of the proposed method over several state-of-the-art approaches in automated image search. The code is now available online.

Efficient similar waveform search using short binary codes obtained through a deep hashing technique

Summary A similar waveform search plays a crucial role in seismology for detecting seismic events, such as small earthquakes and low-frequency events. However, the high computational costs associated with waveform cross-correlation calculations represent bottlenecks during the analysis of long, continuous records obtained from numerous stations. In this study, we developed a deep-learning network to obtain 64-bit hash codes containing information on seismic waveforms. Using this network, we performed a similar waveform search for ∼35 million moving windows developed for the 30 min waveforms recorded continuously at 10 MHz sampling rates using 16 acoustic emission transducers during a laboratory hydraulic fracturing experiment. The sampling points of each channel corresponded to those of the 5.8-year records obtained from typical seismic observations at 100 Hz sampling rates. Of the 35 million windows, we searched for windows with small average Hamming distances among the hash codes of 16 channel waveforms against template hash codes of 6057 events that were catalogued using conventional autoprocessing techniques. The calculation of average Hamming distances is 1430–1530 times faster than that of the corresponding network correlation. This hashing-based template matching enabled the detection of 23,462 additional events. We also demonstrated the feasibility of the hashing-based autocorrelation analysis, where similar event pairs were extracted without templates, by calculating the average Hamming distances for all possible pairs of the ∼35 million windows. This calculation required only 15.5 h under 120 thread parallelisation. This deep hashing approach significantly reduced the required memory compared with locality-sensitive hashing approaches based on random permutations, enabling similar waveform searching on a large-scale dataset.

Incremental image retrieval method based on feature perception and deep hashing

Incremental image retrieval method based on feature perception and deep hashing

Encrypted Video Search with Single/Multiple Writers

Video-based services have become popular. Clients often outsource their videos to the cloud to relieve local maintenance. However, privacy has become a major concern since many videos contain sensitive information. Although retrieving (unencrypted) videos has been extensively investigated, retrieving encrypted multimedia has received relatively rare attention, at best in a limitation of image-based similarity searches. We initiate the study of scalable encrypted video search, enabling clients to query videos similar to an image search. Our modular framework leverages intrinsic attributes of videos, such as semantics and visuals, to effectively capture their contents. We propose a two-step approach whereby lightweight searchable encryption techniques are used for pre-screening, followed by an interactive approach for fine-grained search. Furthermore, we present three instantiations, including one centralized-writer instantiation and two distributed-writer instantiations, to effectively cater to varying needs and scenarios– 1) The centralized one employs forward and backward private searchable encryption [CCS 2017] over deep hashing [CVPR 2020]. 2) Motivated by distributed computing, the multi-writer instantiations building atop HSE [Usenix Security 2022] allows searching the relevant videos contributed by multiple intuitions collaboratively. Our experimental results illustrate their practical performance over multiple real-world datasets, whether in a centralized setting or distributed setting.

Secure speech retrieval method using deep hashing and CKKS fully homomorphic encryption

Secure speech retrieval method using deep hashing and CKKS fully homomorphic encryption

Fabric image retrieval based on decoupling of texture and color feature

Fabric image retrieval, a form of content based image retrieval, is a high value research with the potential to be applied in many fields, such as e-commerce and inventory management. However, this research hotspot is plagued by two major challenges, namely the high requirements for retrieval results and the peculiarities of fabric images. Unlike general image retrieval, fabric image retrieval systems have to pay more attention to texture and color features. To address these challenges, we propose a novel framework for fabric retrieval by using self-supervised and deep hashing techniques. The framework consists of two modules for feature learning and hashing learning. During the feature learning phase, the color and texture information in the image is decoupled under the drive of augmented based pretext tasks. In hashing learning, Bi-half layer is introduced to generate high-quality hash codes. The visualization results indicate that the proposed method performs well for the representation of fabric images. And the experimental results show that the proposed retrieval framework can achieve a good performance (best mAP 0.903) and outperforms other methods, including several deep hashing methods and our previous work.

Unsupervised Dual Deep Hashing with Semantic-Index and Content-Code for Cross-Modal Retrieval.

Hashing technology has exhibited great cross-modal retrieval potential due to its appealing retrieval efficiency and storage effectiveness. Most current supervised cross-modal retrieval methods heavily rely on accurate semantic supervision, which is intractable for annotations with ever-growing sample sizes. By comparison, the existing unsupervised methods rely on accurate sample similarity preservation strategies with intensive computational costs to compensate for the lack of semantic guidance, which causes these methods to lose the power to bridge the semantic gap. Furthermore, both kinds of approaches need to search for the nearest samples among all samples in a large search space, whose process is laborious. To address these issues, this paper proposes an unsupervised dual deep hashing (UDDH) method with semantic-index and content-code for cross-modal retrieval. Deep hashing networks are utilized to extract deep features and jointly encode the dual hashing codes in a collaborative manner with a common semantic index and modality content codes to simultaneously bridge the semantic and heterogeneous gaps for cross-modal retrieval. The dual deep hashing architecture, comprising the head code on semantic index and tail codes on modality content, enhances the efficiency for cross-modal retrieval. A query sample only needs to search for the retrieved samples with the same semantic index, thus greatly shrinking the search space and achieving superior retrieval efficiency. UDDH integrates the learning processes of deep feature extraction, binary optimization, common semantic index, and modality content code within a unified model, allowing for collaborative optimization to enhance the overall performance. Extensive experiments are conducted to demonstrate the retrieval superiority of the proposed approach over the state-of-the-art baselines.

Deep Semantic-Aware Proxy Hashing for Multi-Label Cross-Modal Retrieval

Deep hashing has attracted broad interest in cross-modal retrieval because of its low cost and efficient retrieval benefits. To capture the semantic information of raw samples and alleviate the semantic gap, supervised cross-modal hashing methods that utilize label information which could map raw samples from different modalities into a unified common space, are proposed. Although making great progress, existing deep cross-modal hashing methods are suffering from some problems, such as: 1) Considering multi-label cross-modal retrieval, proxy-based methods ignore the data-to-data relations and fail to explore the combination of the different categories profoundly, which could lead to some samples without common categories being embedded in the vicinity; 2) For feature representation, image feature extractors containing multiple convolutional layers cannot fully obtain global information of images, which results in the generation of sub-optimal binary hash codes. In this paper, by extending the proxy-based mechanism to multi-label cross-modal retrieval, we propose a novel Deep Semantic-aware Proxy Hashing (DSPH) framework, which could embed multi-modal multi-label data into a uniform discrete space and capture fine-grained semantic relations between raw samples. Specifically, by learning multi-modal multi-label proxy terms and multi-modal irrelevant terms jointly, the semantic-aware proxy loss is designed to capture multi-label correlations and preserve the correct fine-grained similarity ranking among samples, alleviating inter-modal semantic gaps. In addition, for feature representation, two transformer encoders are proposed as backbone networks for images and text, respectively, in which the image transformer encoder is introduced to obtain global information of the input image by modeling long-range visual dependencies. We have conducted extensive experiments on three baseline multi-label datasets, and the experimental results show that our DSPH framework achieves better performance than state-of-the-art cross-modal hashing methods. The code for the implementation of our DSPH framework is available at https://github.com/QinLab-WFU/DSPH.

Multi-Relational Deep Hashing for Cross-Modal Search.

Deep cross-modal hashing retrieval has recently made significant progress. However, existing methods generally learn hash functions with pairwise or triplet supervisions, which involves learning the relevant information by splicing partial similarity between data pairs; notably, this approach only captures the data similarity locally and incompletely, resulting in sub-optimal retrieval performance. In this paper, we propose a novel Multi-Relational Deep Hashing (MRDH) approach, which can fully bridge the modality gap by comprehensively modeling the similarity relationship between data in different modalities. In more detail, to investigate the inter-modal relationships, we constrain the consistency of cross-modal pairwise similarities to maintain the semantic similarity across modalities. Moreover, to further capture complete similarity information, we design a new similarity metric, which we term cross-modal global similarity, by encouraging hash codes of similar data pairs from different modalities to approach a common center and hash codes for dissimilar pairs to converge to different centers. Adopting this approach enables our model to generate more discriminative hash codes. Extensive experiments on three benchmark datasets demonstrate the superiority of our method on cross-modal hashing retrieval.

AudioNet: Supervised Deep Hashing for Retrieval of Similar Audio Events

AudioNet: Supervised Deep Hashing for Retrieval of Similar Audio Events

Deep hashing image retrieval based on hybrid neural network and optimized metric learning

While transformers have indeed improved image retrieval accuracy in computer vision, challenges persist, including insufficient and imbalanced feature extraction and the inability to create compact binary codes. This study introduces a novel approach for image retrieval called Vision Transformer with Deep Hashing (VTDH), combining a hybrid neural network and optimized metric learning. Our work offers significant contributions, summarized as follows: We introduce an innovative Strengthened External Attention (NEA) module capable of simultaneous multi-scale feature focus and comprehensive global context assimilation. This enriches the model’s comprehension of both overarching structure and semantics. Additionally, we propose a fresh balanced loss function to tackle the issue of imbalanced positive and negative samples within labels. Notably, this function employs sample labels as input, utilizing the mean value of all sample labels to quantify the frequency gap between positive and negative samples. This approach, combined with a customized balance weight, effectively addresses the challenge of label imbalance. Concurrently, we enhance the quantization loss function, intensifying its penalty for instances where the model’s binary code output surpasses ±1. This reinforcement results in a more robust and stable hash code output. The proposed method is assessed on prominent datasets, including CIFAR-10, NUS-WIDE, and ImageNet. Experimental outcomes reveal superior retrieval accuracy compared to current state-of-the-art techniques. Notably, the VTDH model achieves an exceptional mean average precision (mAP) of 97.3% on the CIFAR-10 dataset.

DMCH: A Deep Metric and Category-Level Semantic Hashing Network for Retrieval in Remote Sensing

The effectiveness of hashing methods in big data retrieval has been proved due to their merit in computational and storage efficiency. Recently, encouraged by the strong discriminant capability of deep learning in image representation, various deep hashing methodologies have emerged to enhance retrieval performance. However, maintaining the semantic richness inherent in remote sensing images (RSIs), characterized by their scene intricacy and category diversity, remains a significant challenge. In response to this challenge, we propose a novel two-stage deep metric and category-level semantic hashing network termed DMCH. First, it introduces a novel triple-selection strategy during the semantic metric learning process to optimize the utilization of triple-label information. Moreover, it inserts a hidden layer to enhance the latent correlation between similar hash codes via a designed category-level classification loss. In addition, it employs additional constraints to keep bit-uncorrelation and bit-balance of generated hash codes. Furthermore, a progressive coarse-to-fine hash code sorting scheme is used for superior fine-grained retrieval and more effective hash function learning. Experiment results on three datasets illustrate the effectiveness and superiority of the proposed method.

Graph-guided deep hashing networks for similar patient retrieval

With the rapid growth and widespread application of electronic health records (EHRs), similar patient retrieval has become an important task for downstream clinical decision support such as diagnostic reference, treatment planning, etc. However, the high dimensionality, large volume, and heterogeneity of EHRs pose challenges to the efficient and accurate retrieval of patients with similar medical conditions to the current case. Several previous studies have attempted to alleviate these issues by using hash coding techniques, improving retrieval efficiency but merely exploring underlying characteristics among instances to preserve retrieval accuracy. In this paper, drug categories of instances recorded in EHRs are regarded as the ground truth to determine the pairwise similarity, and we consider the abundant semantic information within such multi-labels and propose a novel framework named Graph-guided Deep Hashing Networks (GDHN). To capture correlation dependencies among the multi-labels, we first construct a label graph where each node represents a drug category, then a graph convolution network (GCN) is employed to derive the multi-label embedding of each instance. Thus, we can utilize the learned multi-label embeddings to guide the patient hashing process to obtain more informative and discriminative hash codes. Extensive experiments have been conducted on two datasets, including a real-world dataset concerning IgA nephropathy from Peking University First Hospital, and a publicly available dataset from MIMIC-III, compared with traditional hashing methods and state-of-the-art deep hashing methods using three evaluation metrics. The results demonstrate that GDHN outperforms the competitors at different hash code lengths, validating the superiority of our proposal.

Deep Adaptive Quadruplet Hashing With Probability Sampling for Large-Scale Image Retrieval

With the preferable efficiency in storage and computation, hashing has shown potential application in large-scale multimedia retrieval. Compared with traditional hashing algorithms using hand-crafted characteristics, deep hashing inherits the representational capacity of deep neural networks to jointly learn semantic features and hash functions, encoding raw data into compact binary codes with significant discrimination. Generally, most of the current multi-wise hashing methods view the similarity margins between image pairs as constant values in training process. When the distance between sample pairs exceeds the fixed margin, the hashing network would not learn anything. Besides, available hashing methods commonly introduce the random sampling strategy to build training batches and ignore the sample distribution, which is harmful to parameter optimization. In this paper, we propose a novel Deep Adaptive Quadruplet Hashing with probability sampling (DAQH) for discriminative binary code learning. Specifically, with exploring the distribution relationship of raw samples, a non-uniform probability sampling strategy is proposed to build more informative and representative training batches, while maintaining the diversity of training samples. By introducing the prior similarity of sample pairs to calculate corresponding margins, an adaptive margin quadruplet loss is designed to dynamically preserve the underlying semantic relationships with its neighbors. To tune the attributes of binary codes, by combining quadruple regularization and orthogonality optimization, binary code constraint is developed to make the learned embedding with significant discrimination. Extensive experimental results on various benchmark datasets demonstrate our proposed DAQH framework achieves state-of-the-art visual similarity search performance.

Attribute-Aware Deep Hashing With Self-Consistency for Large-Scale Fine-Grained Image Retrieval.

Our work focuses on tackling large-scale fine-grained image retrieval as ranking the images depicting the concept of interests (i.e., the same sub-category labels) highest based on the fine-grained details in the query. It is desirable to alleviate the challenges of both fine-grained nature of small inter-class variations with large intra-class variations and explosive growth of fine-grained data for such a practical task. In this paper, we propose attribute-aware hashing networks with self-consistency for generating attribute-aware hash codes to not only make the retrieval process efficient, but also establish explicit correspondences between hash codes and visual attributes. Specifically, based on the captured visual representations by attention, we develop an encoder-decoder structure network of a reconstruction task to unsupervisedly distill high-level attribute-specific vectors from the appearance-specific visual representations without attribute annotations. Our models are also equipped with a feature decorrelation constraint upon these attribute vectors to strengthen their representative abilities. Then, driven by preserving original entities' similarity, the required hash codes can be generated from these attribute-specific vectors and thus become attribute-aware. Furthermore, to combat simplicity bias in deep hashing, we consider the model design from the perspective of the self-consistency principle and propose to further enhance models' self-consistency by equipping an additional image reconstruction path. Comprehensive quantitative experiments under diverse empirical settings on six fine-grained retrieval datasets and two generic retrieval datasets show the superiority of our models over competing methods. Moreover, qualitative results demonstrate that not only the obtained hash codes can strongly correspond to certain kinds of crucial properties of fine-grained objects, but also our self-consistency designs can effectively overcome simplicity bias in fine-grained hashing.

Deep global semantic structure-preserving hashing via corrective triplet loss for remote sensing image retrieval

With the explosive increase of remote sensing data, how to search for remote sensing data quickly and accurately in a vast dataset is an incredibly critical matter for research subjects. The deep hashing method has become the dominant method for remote sensing image retrieval because of its low-cost storage and high-speed retrieval. However, for the reason of the limitation of fixed convolutional kernels, deep hashing frameworks based on Convolutional Neural Networks (CNNs) fail to obtain the global semantic information well, which leads to the generation of suboptimal solutions. Furthermore, existing hashing methods commonly employ the random sampling strategy or hardest sample mining to build training batches, resulting in bad local minima. To remedy these problems, a novel Deep Global Semantic Structure-preserving Hashing framework via corrective triplet loss (DGSSH) is proposed for remote sensing image retrieval to learn a discriminative and stable embedding space, achieving intra-class confusion and inter-class diversity. Specifically speaking, the feature extraction module based on Swim Transformer architecture is developed to capture global semantic information and multiscale features from remote sensing images. Based on a distribution matching constraint, the corrective triplet loss for deep hashing schemes is designed to reduce the distribution shift caused by the random selection or hardest sample mining. Meanwhile, to reduce the time overhead of the model, the asymmetric learning strategy is employed to perform effective compact representation learning. Numerous experiments have been carried out on three publicly available benchmarks, which indicates that the proposed DGSSH framework could achieve optimal performance for remote sensing image retrieval applications. The source code of our DGSSH framework is hosted at https://github.com/QinLab-WFU/DGSSH.git.

Discriminative Geometric-Structure-Based Deep Hashing for Large-Scale Image Retrieval.

Deep hashing reaps the benefits of deep learning and hashing technology, and has become the mainstream of large-scale image retrieval. It generally encodes image into hash code with feature similarity preserving, that is, geometric-structure preservation, and achieves promising retrieval results. In this article, we find that existing geometric-structure preservation manner inadequately ensures feature discrimination, while improving feature discrimination of hash code essentially determines hash learning retrieval performance. This fact principally spurs us to propose a discriminative geometric-structure-based deep hashing method (DGDH), which investigates three novel loss terms based on class centers to induce the so-called discriminative geometrical structure. In detail, the margin-aware center loss assembles samples in the same class to the corresponding class centers for intraclass compactness, then a linear classifier based on class center serves to boost interclass separability, and the radius loss further puts different class centers on a hypersphere to tentatively reduce quantization errors. An efficient alternate optimization algorithm with guaranteed desirable convergence is proposed to optimize DGDH. We theoretically analyze the robustness and generalization of the proposed method. The experiments on five popular benchmark datasets demonstrate superior image retrieval performance of the proposed DGDH over several state of the arts.

Towards Human–Machine Recognition Alignment: An Adversarilly Robust Multimodal Retrieval Hashing Framework

The multimodality nature of web data has necessitated complex multimodal information retrieval for a wide range of web applications. Deep neural networks (DNNs) have been widely employed to extract semantic features from raw samples to improve retrieval accuracy. In addition, hashing is widely used to improve computational and storage efficiency. As such, deep hashing frameworks have been applied for multimodal retrieval tasks. However, there is still a great recognitive gap between primate brain structure-inspired DNNs and humans. On computer vision tasks, well-crafted DNN models can be easily defeated by invisible small attacks, and this phenomenon indicates a large recognition gap between DNN models and humans. Recently, adversarial defense methods have been shown to improve the human–machine recognition alignment in several classification tasks. However, the robustness problem on the retrieval tasks, especially on the deep hashing-based multimodal retrieval models, is still not well studied. Therefore, in this article, we present an adversarially robust training mechanism to improve model robustness for the purpose of human–machine recognition alignment on retrieval tasks. Through extensive experimental results on several social multimodal retrieval benchmarks, we show that the robust training hashing framework proposed can mitigate the recognition gap on retrieval tasks. Our study highlights the necessity of robustness enhancement on deep hashing models.

Multi-Scale Feature Fusion Based on PVTv2 for Deep Hash Remote Sensing Image Retrieval

For high-resolution remote sensing image retrieval tasks, single-scale features cannot fully express the complexity of the image information. Due to the large volume of remote sensing images, retrieval requires extensive memory and time. Hence, the problem of how to organically fuse multi-scale features and enhance retrieval efficiency is yet to be resolved. We propose an end-to-end deep hash remote sensing image retrieval model (PVTA_MSF) by fusing multi-scale features based on the Pyramid Vision Transformer network (PVTv2). We construct the multi-scale feature fusion module (MSF) by using a global attention mechanism and a multi-head self-attention mechanism to reduce background interference and enhance the representation capability of image features. Deformable convolution is introduced to address the challenge posed by varying target orientations. Moreover, an intra-class similarity (ICS) loss is proposed to enhance the discriminative capability of the hash feature by minimizing the distance among images of the same category. The experimental results show that, compared with other state-of-the-art methods, the proposed hash feature could yield an excellent representation of remote sensing images and improve remote sensing image retrieval accuracy. The proposed hash feature can gain an increase of 4.2% and 1.6% in terms of mAP on the UC Merced and NWPU-RESISC45 datasets, respectively, in comparison with other methods.

Deep attention sampling hashing for efficient image retrieval

Hashing has received broad attention in large-scale image retrieval due to its appealing efficiency in computation and storage. Particularly, with the drawn of deep learning, much efforts have been directed towards using deep neural networks to learn feature representations and hash codes simultaneously, and the developed deep hashing methods have shown superior performance over conventional hashing methods. In this paper, we propose Deep Attention Sampling Hashing (DASH), a novel deep hashing method that yields high-quality hash codes to enable efficient image retrieval. Specifically, we employ two sub-networks in DASH, i.e., a master branch and a part branch, to capture global structure features and discriminative feature representations, respectively. Furthermore, we develop an Attention Sampler Module (ASM), which consists of an Object Region Extraction (ORE) block and an Informative Patch Generation (IPG) block, to yield richer informative image patches. The ORE block provides a well-designed multi-scale attentional fusion mechanism to highlight and extract the significant regions of images, and the IPG block employs a direction-specific shift mechanism to generate desired image patches with discriminative details. Both blocks could be seamlessly integrated into various convolutional neural network (CNN) architectures. Subsequently, we conduct knowledge distillation optimization to transfer the details learned by the part branch into the master branch to guide hash code learning. In addition, we design a Weibull quantization loss to minimize the information loss caused by binary quantization. The experimental results on three benchmark datasets demonstrate the effectiveness of the proposed DASH with respect to different evaluation metrics.