Binary Hash Codes Research Articles

Single-loss hash image retrieval method based on improved visual transformer

Deep hashing methods have gained popularity in image retrieval due to their advantages such as low storage requirements and high efficiency. However, existing deep hashing methods for large-scale image retrieval tasks suffer from issues including low discriminative power of binary hash codes, difficult optimization of losses, and low retrieval accuracy. This paper proposes a single-loss hash image retrieval method based on an improved visual transformer to address these issues. The proposed method utilizes a pre-trained Vision Transformer (ViT) on ImageNet as the backbone network, augmented with a hash coding layer to extract image features more comprehensively. Additionally, we design a single learning objective loss function that addresses the discriminative power of hash codes and quantization errors, thereby eliminating the complexity of adjusting various loss weights. Experimental evaluations on ImageNet100, NUS-WIDE, CIFAR10, and MS-COCO datasets demonstrate the superior performance of the proposed method compared to contemporary methods, indicating its adaptability to diverse data.

Semantic-Preserving Surgical Video Retrieval With Phase and Behavior Coordinated Hashing.

Medical professionals rely on surgical video retrieval to discover relevant content within large numbers of videos for surgical education and knowledge transfer. However, the existing retrieval techniques often fail to obtain user-expected results since they ignore valuable semantics in surgical videos. The incorporation of rich semantics into video retrieval is challenging in terms of the hierarchical relationship modeling and coordination between coarse- and fine-grained semantics. To address these issues, this paper proposes a novel semantic-preserving surgical video retrieval (SPSVR) framework, which incorporates surgical phase and behavior semantics using a dual-level hashing module to capture their hierarchical relationship. This module preserves the semantics in binary hash codes by transforming the phase and behavior similarities into high- and low-level similarities in a shared Hamming space. The binary codes are optimized by performing a reconstruction task, a high-level similarity preservation task, and a low-level similarity preservation task, using a coordinated optimization strategy for efficient learning. A self-supervised learning scheme is adopted to capture behavior semantics from video clips so that the indexing of behaviors is unencumbered by fine-grained annotation and recognition. Experiments on four surgical video datasets for two different disciplines demonstrate the robust performance of the proposed framework. In addition, the results of the clinical validation experiments indicate the ability of the proposed method to retrieve the results expected by surgeons. The code can be found at https://github.com/trigger26/SPSVR.

Multi-Modal Hashing for Efficient Multimedia Retrieval: A Survey

With the explosive growth of multimedia contents, multimedia retrieval is facing unprecedented challenges on both storage cost and retrieval speed. Hashing technique can project the high-dimensional data into compact binary hash codes. With it, the most time-consuming semantic similarity computation during the multimedia retrieval process can be significantly accelerated with fast Hamming distance computation, and meanwhile the storage cost can be reduced greatly by the binary embedding. In the light of this, multi-modal hashing has recently received considerable attention to support large-scale multimedia retrieval. Different from uni-modal hashing, the multi-modal hashing focuses on modeling the multi-modal semantics and further preserving them into binary hash codes with hash learning. In this paper, we first systematically review the existing learning to hash methods for efficient multimedia retrieval, categorizing them according to the multimedia retrieval tasks, the specific multi-modal semantic modeling techniques, and hash learning strategies. Thereafter, we present the performance comparison results. We ultimately discuss the challenges and potential research directions that may require further investigation in multi-modal hash learning. To facilitate the research on multi-modal hashing, we develop an open-source performance comparison tool at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/BMC-SDNU/Hashing-Retrieval</uri> .

Deep Semantic-Aware Proxy Hashing for Multi-Label Cross-Modal Retrieval

Deep hashing has attracted broad interest in cross-modal retrieval because of its low cost and efficient retrieval benefits. To capture the semantic information of raw samples and alleviate the semantic gap, supervised cross-modal hashing methods that utilize label information which could map raw samples from different modalities into a unified common space, are proposed. Although making great progress, existing deep cross-modal hashing methods are suffering from some problems, such as: 1) Considering multi-label cross-modal retrieval, proxy-based methods ignore the data-to-data relations and fail to explore the combination of the different categories profoundly, which could lead to some samples without common categories being embedded in the vicinity; 2) For feature representation, image feature extractors containing multiple convolutional layers cannot fully obtain global information of images, which results in the generation of sub-optimal binary hash codes. In this paper, by extending the proxy-based mechanism to multi-label cross-modal retrieval, we propose a novel Deep Semantic-aware Proxy Hashing (DSPH) framework, which could embed multi-modal multi-label data into a uniform discrete space and capture fine-grained semantic relations between raw samples. Specifically, by learning multi-modal multi-label proxy terms and multi-modal irrelevant terms jointly, the semantic-aware proxy loss is designed to capture multi-label correlations and preserve the correct fine-grained similarity ranking among samples, alleviating inter-modal semantic gaps. In addition, for feature representation, two transformer encoders are proposed as backbone networks for images and text, respectively, in which the image transformer encoder is introduced to obtain global information of the input image by modeling long-range visual dependencies. We have conducted extensive experiments on three baseline multi-label datasets, and the experimental results show that our DSPH framework achieves better performance than state-of-the-art cross-modal hashing methods. The code for the implementation of our DSPH framework is available at https://github.com/QinLab-WFU/DSPH.

Multimodal Mutual Information Maximization: A Novel Approach for Unsupervised Deep Cross-Modal Hashing.

In this article, we adopt the maximizing mutual information (MI) approach to tackle the problem of unsupervised learning of binary hash codes for efficient cross-modal retrieval. We proposed a novel method, dubbed cross-modal info-max hashing (CMIMH). First, to learn informative representations that can preserve both intramodal and intermodal similarities, we leverage the recent advances in estimating variational lower bound of MI to maximizing the MI between the binary representations and input features and between binary representations of different modalities. By jointly maximizing these MIs under the assumption that the binary representations are modeled by multivariate Bernoulli distributions, we can learn binary representations, which can preserve both intramodal and intermodal similarities, effectively in a mini-batch manner with gradient descent. Furthermore, we find out that trying to minimize the modality gap by learning similar binary representations for the same instance from different modalities could result in less informative representations. Hence, balancing between reducing the modality gap and losing modality-private information is important for the cross-modal retrieval tasks. Quantitative evaluations on standard benchmark datasets demonstrate that the proposed method consistently outperforms other state-of-the-art cross-modal retrieval methods.

Sequential and Correlated Image Hash Code Generation with Deep Reinforcement Learning

Image hashing is an algorithm used to represent an image with a unique value. Hashing methods, which are generally developed to search for similar examples of an image, have gained a new dimension with the use of deep network structures and better results have started to be obtained with the methods. The developed deep network models generally consider hash functions independently and do not take into account the correlation between them. In addition, most of the existing data-dependent hashing methods use pairwise/triplet similarity metrics that capture data relationships from a local perspective. In this study, the Central similarity metric, which can achieve better results, is adapted to the deep reinforcement learning method with sequential learning strategy, and successful results are obtained in learning binary hash codes. By taking into account the errors of previous hash functions in the deep reinforcement learning strategy, a new model is presented that performs interrelated and central similarity based learning.

Gaze-guided CT image retargeting by multi-attribute binary hashing

Computed tomography (CT) imaging is pervasively utilized for detecting tumors and internal body injuries. CT image retargeting means to horizontally/vertically shrink the semantically non-salient regions (e.g, the normal organs) while preserving the salient ones (e.g., the diseased organs) inside a CT image, as exemplified in Fig. 1. In practice, retargeting can substantially facilitate CT image displaying, which can benefit the subsequent medical treatment. In this work, we propose a bio-inspired CT image retargeting pipeline by mimicking human gaze behavior. More specifically, for each CT image, we extract the gaze shifting path (GSP) to capture human gaze distribution during the visual perception toward each CT image. Afterward, a multi-attribute binary hashing (MABH) is formulated to exploit the semantics of these GSPs. Thereby, each graphlet can be converted into the binary hash codes. Finally, the hash codes corresponding to GSP from each CT image are quantized into a feature vector, which is leveraged to learn a Gaussian mixture model (GMM) that guides CT image shrinking. In the experiments, to evaluate how gaze allocation influencing CT image retargeting, a user study is designed to compare the GSPs produced by normal observers and Alzheimer’s patients respectively. Besides, a comparative study has verified the superiority of our method.

One for more: Structured Multi-Modal Hashing for multiple multimedia retrieval tasks

Hashing has been widely studied in support of efficient multimedia retrieval due to its high computation and storage efficiency. However, existing multimedia hashing models are designed only for specific multimedia retrieval tasks, e.g., cross-modal retrieval and composite multi-modal retrieval. To simultaneously support multiple multimedia retrieval tasks, multiple different hashing models should be equipped in a multimedia retrieval system. This paper proposes a Structured Multi-Modal Hashing (SMMH) method, which can learn only a hash model for multiple multimedia retrieval tasks. Specifically, we generate structured hash codes by independently projecting the modality data into the Hamming space, where each part of hash codes preserves the specific characteristic of the specific modality data, and they are collaborated to represent the whole multi-modal data. To bridge the heterogeneous gap between multiple modalities, a fusion multi-modal space is constructed based on structured hash codes with label and pair-wise semantic supervision. Besides, we learn hash functions for composite multi-modal retrieval by projecting heterogeneous modalities into the collaborative multi-modal features. Finally, we develop a discrete hash optimization strategy to efficiently solve binary hash codes. Extensive experiments show that our model outperforms the strong baseline methods on multiple multimedia retrieval tasks. Source codes are available at https://github.com/ChaoqunZheng/SMMH.

An enhanced noise-tolerant hashing for drone object detection

Drone, a.k.a. Unmanned aerial vehicle (UAV), has been pervasively applied in geological hazard monitoring, smart agriculture, and urban planning in the past decade. In this work, we fuse multiple attributes into a noise-tolerant hashing framework that can detect objects from drone pictures extremely fast. Our method can intrinsically and flexibly encode various topological structures from each target object, based on which multi-scale objects can be discovered in a view- and altitude-invariant way. Moreover, by leveraging lF and l1 norms collaboratively, the calculated hash codes are robust to low quality drone pictures and noisy semantic labels. More specifically, for each drone-borne picture, we extract visually/semantically salient object parts inside it. To characterize their topological structure, we construct a graphlet by linking the spatially adjacent object patches into a small graph. Subsequently, a binary matrix factorization (MF) is designed to hierarchically exploit the semantics of these graphlets, wherein three attributes: i) deep binary hash codes learning, ii) contaminated pictures/labels denoising, and iii) adaptive data graph updating are seamlessly incorporated. Such multi-attribute binary MF can be solved iteratively, and in turn each graphlet is transformed into the binary hash codes. Finally, the hash codes corresponding to graphlets within each drone photo are utilized for ranking-based object discovery. Comprehensive experiments on the DAC-SDC, MOHR, and our self-compiled data set have demonstrated the competitively speed and accuracy of our method. As a byproduct, we employ an elaborately-designed FPGA architecture to calculate our hash codes. On average, a 57 frames per second (fps) object detection speed is achieved on 4K drone videos (without temporal modeling).

Unsupervised soft-to-hard hashing with contrastive learning

In order to optimize a binary hash code with deep neural networks, greedy optimization with the discrete constraint relaxation is widely used due to its efficiency. However, noises from skipping gradients may destabilize the optimization in contrastive learning since the noises are superimposed on embedding from two distorted views. To reduce such undesirable impact from the noises, we propose a novel soft-to-hard hashing method by introducing an auxiliary loss in the penultimate block of the model based on the information bottleneck principle, which not only propagates the correlations between positives but also makes the embedding to be non-redundant to the samples in a high-dimensional continuous space. Our method beats the state-of-the-art result on three public datasets and demonstrates how soft code can help greedy back-propagation find a better solution during optimization. Another benefit of our method is that it enables a joint training of unsupervised representational learning and hash code generation, and obtains 5.4% mAP gain over conventional step-by-step training on the CIFAR-10 dataset.

Non-Relaxation Deep Hashing Method for Fast Image Retrieval

Deep hashing methods utilize an end-to-end framework to mutually learn feature representations and hash codes, thereby achieving a better retrieval performance. Traditional supervised hashing methods adopt handcrafted features for hashing function learning and then generate hash codes through classification and quantization. The lack of adaptability and independence of the quantization procedure leads to low retrieval accuracy of supervised hashing methods with handcrafted features in image retrieval. In this study, a non-relaxation deep hashing method for fast image retrieval is proposed. In this method, a differentiable host thresholding function is used to encourage hash-like codes to approach -1 or 1 non-linearly at the output of the convolutional neural, instead of the symbol function for quantization used in the traditional method. The output of the host thresholding function is directly used to compute the network training error, and a loss function is elaborately designed with the norm to constrain each bit of the hash-like code to be as binary as possible. Finally, a symbol function is added outside the trained network model to generate binary hash codes for image storage and retrieval in a low-dimensional binary space. Extensive experiments on two large-scale public datasets show that our method can effectively learn image features, generate accurate binary hash codes, and outperform state-of-the-art methods in terms of the mean average precision.

Unsupervised Adaptive Feature Selection with Binary Hashing.

Unsupervised feature selection chooses a subset of discriminative features to reduce feature dimension under the unsupervised learning paradigm. Although lots of efforts have been made so far, existing solutions perform feature selection either without any label guidance or with only single pseudo label guidance. They may cause significant information loss and lead to semantic shortage of the selected features as many real-world data, such as images and videos are generally annotated with multiple labels. In this paper, we propose a new Unsupervised Adaptive Feature Selection with Binary Hashing (UAFS-BH) model, which learns binary hash codes as weakly-supervised multi-labels and simultaneously exploits the learned labels to guide feature selection. Specifically, in order to exploit the discriminative information under the unsupervised scenarios, the weakly-supervised multi-labels are learned automatically by specially imposing binary hash constraints on the spectral embedding process to guide the ultimate feature selection. The number of weakly-supervised multi-labels (the number of "1" in binary hash codes) is adaptively determined according to the specific data content. Further, to enhance the discriminative capability of binary labels, we model the intrinsic data structure by adaptively constructing the dynamic similarity graph. Finally, we extend UAFS-BH to multi-view setting as Multi-view Feature Selection with Binary Hashing (MVFS-BH) to handle the multi-view feature selection problem. An effective binary optimization method based on the Augmented Lagrangian Multiple (ALM) is derived to iteratively solve the formulated problem. Extensive experiments on widely tested benchmarks demonstrate the state-of-the-art performance of the proposed method on both single-view and multi-view feature selection tasks. For the purpose of reproducibility, we provide the source codes and testing datasets at https://github.com/shidan0122/UMFS.git..

Regularizing Binary Neural Networks via Ensembling for Efficient Person Re-Identification

This study aims to leverage Binary Neural Networks (BNN) to learn binary hash codes for efficient person re-identification (ReID). BNNs, which use binary weights and activations, show promise in speeding up the inference time in deep models. However, BNNs typically suffer from performance degradation mainly due to the discontinuity of the binarization operation. Proxy functions have been proposed to calculate the gradients in the backward propagation, but they lead to the gradient mismatch problem. In this study, we propose to address the gradient mismatch problem by designing a multi-branch ensemble model consisting of many weak hash code learners. Specifically, our design aggregates the gradients from multiple branches, which allows a better approximation of the gradients and regularizes the network. Our model adds little computational cost to the baseline BNN since a vast amount of network parameters are shared between the weak learners. Combining the efficiency of the BNNs and hash code learning, we obtain an effective ensemble model which is efficient both in feature extraction and ranking phases. Our experiments demonstrate that the proposed model outperforms a single BNN by more than %20 using nearly the same amount of floating point operations. Moreover, the proposed model outperforms a conventional ensemble of BNN by more than %7 while being nearly 10x and 2x more efficient in terms of CPU consumption and memory footprint, respectively. We explore the performance of BNNs for efficient person ReID as one of the first systems available in the literature. Moreover, we adopt the proposed ensemble model for further validation of the image classification task and observe that our method effectively regularizes BNNs, providing robustness to hyperparameter selection and producing more consistent results under different settings.

Encrypting Hashing Against Localization

Hashing for localization (HfL) is an effective method for fast localizing specific scenes in a large-scale remote sensing image. Key to its efficiency arises from a comprehensive deep hashing network that generates representational binary hash codes for image patches cropped from the remote sensing image. On the other hand, this paper will investigate the problem of encrypting the remote sensing image against the HfL task. We refer to the new task as encrypting hashing against localization (EHaL). We characterize the EHaL task in term of two cues: (I) An encrypted image patch is supposed to appear as visually similar to its original image patch as possible; (II) The hash code generated by the deep hashing network for the encrypted image patch is supposed to be not close to its original class but close to a different class. Following the two cues, we develop an encrypted patch generator, which is trained in an adversarial fashion. Based on the encrypted patch generator, we propose two remote sensing image encryption frameworks that can cause non-localization and mis-localization to the HfL task separately. Experiments validates the effectiveness of our method. Reproducible executions are given at https://github.com/JingpengHan/EHaL.

RelaHash: Deep Hashing With Relative Position

Deep hashing has been widely used as a solution to encoding binary hash code for approximating nearest neighbor problem. It has been showing superior performance in terms of its ability to index high-level features by learning compact binary code. Many recent state-of-the-art deep hashing methods often use multiple loss terms at once, thus introducing optimization difficulty and may result in sub-optimal hash codes. OrthoHash [1] was proposed to replace those losses with just a single loss function. However, the quantization error minimization problem in OrthoHash is still not addressed effectively. In this paper, we take one step further - propose a single-loss model that can effectively minimize the quantization error without explicit loss terms. Specifically, we introduce a new way to measure the similarity between the relaxed codes with centroids, called relative similarity. The relative similarity is the similarity between the relative position representation of continuous codes and the normalized centroids. The resulting model outperforms many state-of-the-art deep hashing models on popular benchmark datasets.

Deep Feature Pyramid Hashing for Efficient Image Retrieval

Thanks to the success of deep learning, deep hashing has recently evolved as a leading method for large-scale image retrieval. Most existing hashing methods use the last layer to extract semantic information from the input image. However, these methods have deficiencies because semantic features extracted from the last layer lack local information, which might impact the global system’s performance. To this end, a Deep Feature Pyramid Hashing DFPH is proposed in this study, which can fully utilize images’ multi-level visual and semantic information. Our architecture applies a new feature pyramid network designed for deep hashing to the VGG-19 model, so the model becomes able to learn the hash codes from various feature scales and then fuse them to create final binary hash codes. The experimental results performed on two widely used image retrieval datasets demonstrate the superiority of our method.

Hybrid-attention based Feature-reconstructive Adversarial Hashing Networks for Cross-modal Retrieval

With the massive growth of data of various modal types, people no longer use a single modal retrieval method, but a cross-modal retrieval method when performing retrieval tasks. Such methods often need to store data efficiently while maintaining the characteristics of fast query. Because the hashing learning method can represent the original high-dimensional data through a simple and compact binary hash code, which can greatly compress the data size and facilitate data storage and mutual retrieval, the cross-modal hashing retrieval has gradually become a hot topic in recent years. However, how to bridge the gap between modalities to improve the retrieval performance further is still a challenging problem. In order to solve this problem, we propose a Hybrid-attention based Feature-reconstructive Adversarial Hashing (HFAH) networks for cross-modal retrieval. First, a label semantic guidance module is introduced to guide the extraction process of text features and image features through the learning of labels, so as to fully maintain the semantic similarity between different modal data. Then, the hybrid-attention module is introduced to make the extracted data contain richer semantic information. Finally, the feature reconstruction network is used to make the relevant degree between similar cross-modal data pairs higher than that between dissimilar data pairs. Related experiments on two benchmark datasets confirm to us that HFAH performs better than several existing cross- modal retrieval methods.

Cross-Modal Contrastive Hashing Retrieval for Infrared Video and EEG.

It is essential to estimate the sleep quality and diagnose the clinical stages in time and at home, because they are closely related to and important causes of chronic diseases and daily life dysfunctions. However, the existing "gold-standard" sensing machine for diagnosis (Polysomnography (PSG) with Electroencephalogram (EEG) measurements) is almost infeasible to deploy at home in a "ubiquitous" manner. In addition, it is costly to train clinicians for the diagnosis of sleep conditions. In this paper, we proposed a novel technical and systematic attempt to tackle the previous barriers: first, we proposed to monitor and sense the sleep conditions using the infrared (IR) camera videos synchronized with the EEG signal; second, we proposed a novel cross-modal retrieval system termed as Cross-modal Contrastive Hashing Retrieval (CCHR) to build the relationship between EEG and IR videos, retrieving the most relevant EEG signal given an infrared video. Specifically, the CCHR is novel in the following two perspectives. Firstly, to eliminate the large cross-modal semantic gap between EEG and IR data, we designed a novel joint cross-modal representation learning strategy using a memory-enhanced hard-negative mining design under the framework of contrastive learning. Secondly, as the sleep monitoring data are large-scale (8 h long for each subject), a novel contrastive hashing module is proposed to transform the joint cross-modal features to the discriminative binary hash codes, enabling the efficient storage and inference. Extensive experiments on our collected cross-modal sleep condition dataset validated that the proposed CCHR achieves superior performances compared with existing cross-modal hashing methods.

Learning Binary Hash Codes Based on Adaptable Label Representations.

The goal of supervised hashing is to construct hash mappings from collections of images and semantic annotations such that semantically relevant images are embedded nearby in the learned binary hash representations. Existing deep supervised hashing approaches that employ classification frameworks with a classification training objective for learning hash codes often encode class labels as one-hot or multi-hot vectors. We argue that such label encodings do not well reflect semantic relations among classes and instead, effective class label representations ought to be learned from data, which could provide more discriminative signals for hashing. In this article, we introduce Adaptive Labeling Deep Hashing (AdaLabelHash) that learns binary hash codes based on learnable class label representations. We treat the class labels as the vertices of a K -dimensional hypercube, which are trainable variables and adapted together with network weights during the backward network training procedure. The label representations, referred to as codewords, are the target outputs of hash mapping learning. In the label space, semantically relevant images are then expressed by the codewords that are nearby regarding Hamming distances, yielding compact and discriminative binary hash representations. Furthermore, we find that the learned label representations well reflect semantic relations. Our approach is easy to realize and can simultaneously construct both the label representations and the compact binary embeddings. Quantitative and qualitative evaluations on several popular benchmarks validate the superiority of AdaLabelHash in learning effective binary codes for image search.

Music Recognition Using Blockchain Technology and Deep Learning.

The purposes are to recognize and classify different music characteristics and strengthen the copyright protection system for original digital music in the big data era. Deep learning (DL) and blockchain technology are applied and researched herein. Based on CNN (Convolutional Neural Network), a music recognition method combined with hashing learning is proposed. The error generated when outputting the binary hash code is considered, and the semantic similarity of the hash code is ensured. Besides, the application of blockchain technology in the current intellectual property protection in original music is discussed. According to digital music property rights protection needs, the system is divided into modules, and its functions are designed. The system ensures its various functions by applying the application protocol designed in the Algor and network. In the experiments, the MagnaTagATune dataset is selected to verify the performance of the proposed CRNNH (Convolutional Recurrent Neural Network Hashing) algorithm. The algorithm shows the best music recognition performance under different bit numbers. When the number of connections is about 100, the QPS value of the blockchain-based music property rights protection system can be stabilized at about 20,000. At any number of threads, the system pressure will increase dramatically with the increase in the number of analog connections. The music recognition algorithm based on DL and hash method discussed is of great significance in improving the classification accuracy of music recognition. The application of blockchain technology in the copyright protection platform of original music works can protect the copyright of digital music and ensure the operation performance of the system.