Scalable Image Retrieval Research Articles

Cross-modality sub-image retrieval using contrastive multimodal image representations

In tissue characterization and cancer diagnostics, multimodal imaging has emerged as a powerful technique. Thanks to computational advances, large datasets can be exploited to discover patterns in pathologies and improve diagnosis. However, this requires efficient and scalable image retrieval methods. Cross-modality image retrieval is particularly challenging, since images of similar (or even the same) content captured by different modalities might share few common structures. We propose a new application-independent content-based image retrieval (CBIR) system for reverse (sub-)image search across modalities, which combines deep learning to generate representations (embedding the different modalities in a common space) with robust feature extraction and bag-of-words models for efficient and reliable retrieval. We illustrate its advantages through a replacement study, exploring a number of feature extractors and learned representations, as well as through comparison to recent (cross-modality) CBIR methods. For the task of (sub-)image retrieval on a (publicly available) dataset of brightfield and second harmonic generation microscopy images, the results show that our approach is superior to all tested alternatives. We discuss the shortcomings of the compared methods and observe the importance of equivariance and invariance properties of the learned representations and feature extractors in the CBIR pipeline. Code is available at: https://github.com/MIDA-group/CrossModal_ImgRetrieval.

Towards Codebook-Free Deep Probabilistic Quantization for Image Retrieval.

As a classical feature compression technique, quantization is usually coupled with inverted indices for scalable image retrieval. Most quantization methods explicitly divide feature space into Voronoi cells, and quantize feature vectors in each cell into the centroids learned from data distribution. However, Voronoi decomposition is difficult to achieve discriminative space partition for semantic image retrieval. In this paper, we explore semantic-aware feature space partition by deep neural network instead of Voronoi cells. To this end, we propose a new deep probabilistic quantization method, abbreviated as DeepIndex, which constructs inverted indices without explicit centroid learning. In our method, the deep neural network takes an image as input and outputs its probability of being put into each inverted index list. During training, we progressively quantize each image into the inverted lists with the top- T maximal probabilities, and calculate the reward of each trial based on retrieval accuracy. We optimize the deep neural network to maximize the probability of the inverted list with maximal reward. In this way, the retrieval performance is directly optimized, leading to a more semantically discriminative space partition than other quantization methods. The experiments on public image datasets demonstrate the effectiveness of our DeepIndex method on semantic image retrieval.

Classification on Unsupervised Deep Hashing With Pseudo Labels Using Support Vector Machine for Scalable Image Retrieval

The content-based image retrieval (CBIR) method operates on the low-level visual features of the user input query object, which makes it difficult for users to formulate the query and also does not provide adequate retrieval results. In the past, image annotation was suggested as the best possible framework for CBIR, which works on automatically signing keywords to images that support image retrieval. The recent successes of deep learning techniques, especially Convolutional Neural Networks (CNN), in solving computer vision applications have inspired me to work on this paper to solve the problem of CBIR using a dataset of annotated images

FhVLAD: Fine-grained quantization and encoding high-order descriptor statistics for scalable image retrieval

We are interested in the encoding of local descriptors of an image (e.g. SIFT) to design a compact representation vector and thereby address scalable image retrieval. We revisit the implicit design choices in the popular vector of locally aggregated descriptors (VLAD), which aggregates the residuals of descriptors to the codewords. VLAD’s use of a coarse codebook and first-order descriptor statistics in residual computation results in less discriminative residuals. To address this problem, we propose a division of codebook feature space using a novel fine-grained quantization strategy. After quantization, we embed the resulting residuals with high-order statistics of descriptor distribution. Experiments on three challenging image retrieval datasets (INRIA Holidays, UKBench, Oxford 5k) confirm the improved discriminative power of our novel encoding method called FhVLAD. We observe superior accuracy to baseline and competitive performance to state-of-the-art techniques with a limited increase in dimension.

Zero-Shot Learning to Index on Semantic Trees for Scalable Image Retrieval.

In this study, we develop a new approach, called zero-shot learning to index on semantic trees (LTI-ST), for efficient image indexing and scalable image retrieval. Our method learns to model the inherent correlation structure between visual representations using a binary semantic tree from training images which can be effectively transferred to new test images from unknown classes. Based on predicted correlation structure, we construct an efficient indexing scheme for the whole test image set. Unlike existing image index methods, our proposed LTI-ST method has the following two unique characteristics. First, it does not need to analyze the test images in the query database to construct the index structure. Instead, it is directly predicted by a network learnt from the training set. This zero-shot capability is critical for flexible, distributed, and scalable implementation and deployment of the image indexing and retrieval services at large scales. Second, unlike the existing distance-based index methods, our index structure is learnt using the LTI-ST deep neural network with binary encoding and decoding on a hierarchical semantic tree. Our extensive experimental results on benchmark datasets and ablation studies demonstrate that the proposed LTI-ST method outperforms existing index methods by a large margin while providing the above new capabilities which are highly desirable in practice.

Sparse graph based self-supervised hashing for scalable image retrieval

In recent years, learning-based image hashing techniques have elicited wide interest among researchers because they can be applied in high-dimensional data such as videos and images. Supervised hashing techniques can achieve a satisfactory retrieval performance, but are excessively reliant on label information, resulting in the appearance of unsupervised hashing methods. However, existing unsupervised hashing methods often fail to obtain effective similarity information from the training set; this causes a large degradation in their retrieval performance. Some pseudo label-based methods partially alleviate this problem, but are sensitive to the pre-defined number of categories. Therefore, in this paper, we mainly discuss a solution to the above-mentioned problems regarding image hashing. We propose a sparse graph based self-supervised hashing method in which a sparse graph is constructed, which not only circumvents the requirement of a predefined number of categories as compared with pseudo label-based methods, but also significantly reduces the memory demand compared with a dense graph-based method. In addition, we also exploit a self-supervised reconstruction constraint to further preserve the semantic information. These items are combined in a linear manner and optimized using an iterative strategy. Four representative datasets, including two single-label and two multi-label datasets, are employed to evaluate our method. The results, based on multiple metrics, show that our method can outperform other state-of-the-art methods.

Unsupervised deep triplet hashing with pseudo triplets for scalable image retrieval

Deep learning based hashing methods have been proven to be effective in the field of image retrieval recently. Among them, most high-performance methods are supervised frameworks, which require annotated labels by humans. Considering the difficulty of labeling large-scale image datasets, unsupervised methods, which just need images themselves for training, are more suitable for practical applications. However, how to improve the discriminative ability of hash codes generated by unsupervised models still remains as a challenging problem. In this paper, we present a novel deep framework called Unsupervised Deep Triplet Hashing (UDTH) for scalable image retrieval. UDTH builds pseudo triplets based on the neighborhood structure in the high-dimensional visual feature space, and then solves two problems through the proposed objective function: 1) Triplet network is utilized to maximize the distance between different classes of binary representation; 2) Autoencoder and Binary quantization are exploited to learn hash codes which maintain the structural information of original samples. Extensive experiments on the datasets of CIFAR-10, NUS-WIDE and MIRFLICKR-25K are conducted, and the results show that our proposed UDTH is superior to the state-of-the-art methods.

The Scalable Image Retrieval Systems and Applications

Advances in information storage and image acquisition technologies have enabled the creation of enormous image datasets. during this situation, it's necessary to develop applicable data systems to with efficiency manage these collections. the most typical approaches use the supposed Content-Based Image Retrieval (CBIR) systems. Basically, these systems attempt to retrieve pictures like a user-defined specification or pattern (e.g., form sketch, image example). Their goal is to support image retrieval supported content properties (e.g., shape, color, texture), typically encoded into feature vectors. one among the most benefits of the CBIR approach is that the chance of AN automatic retrieval method, rather than the standard keyword-based approach, thattypically needs terribly toilsome and long previous annotation of info pictures. The CBIR technology has been utilized in many applications like fingerprint identification, variety data systems, digital libraries, crime bar, medicine, historical analysis, among others.

Combining Convolutional Neural Network and Markov Random Field for Semantic Image Retrieval

With the rapidly growing number of images over the Internet, efficient scalable semantic image retrieval becomes increasingly important. This paper presents a novel approach for semantic image retrieval by combining Convolutional Neural Network (CNN) and Markov Random Field (MRF). As a key step, image concept detection, that is, automatically recognizing multiple semantic concepts in an unlabeled image, plays an important role in semantic image retrieval. Unlike previous work that uses single-concept classifiers one by one, we detect semantic multiconcept by using a multiconcept scene classifier. In other words, our approach takes multiple concepts as a holistic scene for multiconcept scene learning. Specifically, we first train a CNN as a concept classifier, which further includes two types of classifiers: a single-concept fully connected classifier that is best suited to single-concept detection and a multiconcept scene fully connected classifier that is good for holistic scene detection. Then we propose an MRF-based late fusion approach that is able to effectively learn the semantic correlation between the single-concept classifier and multiconcept scene classifier. Finally, the semantic correlation among the subconcepts of images is cought to further improve detection precision. In order to investigate the feasibility and effectiveness of our proposed approach, we conduct comprehensive experiments on two publicly available image databases. The results show that our proposed approach outperforms several state-of-the-art approaches.

Semantic facial scores and compact deep transferred descriptors for scalable face image retrieval

Abstract Face retrieval systems intend to locate index of identical faces to a given query face. The performance of these systems heavily relies on the careful analysis of different facial attributes (gender, race, etc.) since these attributes can tolerate some degrees of geometrical distortion, expressions, and occlusions. However, solely employing facial scores fail to add scalability. Besides, owing to the discriminative power of CNN (convolutional neural network) features, recent works have employed a complete set of deep transferred CNN features with a large dimensionality to obtain enhancement; yet these systems require high computational power and are very resource-demanding. This study aims to exploit the distinctive capability of semantic facial attributes while their retrieval results are refined by a proposed subset feature selection to reduce the dimensionality of the deep transferred descriptors. The constructed compact deeply transferred descriptors (CDTD) not only have a largely reduced dimension but also more discriminative power. Lastly, we have proposed a new performance metric which is tailored for the case of face retrieval called ImAP (Individual Mean Average Precision) and is used to evaluate retrieval results. Multiple experiments on two scalable face datasets demonstrated the superior performance of the proposed CDTD model outperforming state-of-the-art face retrieval results.

Unsupervised Deep Hashing With Pseudo Labels for Scalable Image Retrieval.

In order to achieve efficient similarity searching, hash functions are designed to encode images into low-dimensional binary codes with the constraint that similar features will have a short distance in the projected Hamming space. Recently, deep learning-based methods have become more popular, and outperform traditional non-deep methods. However, without label information, most state-of-the-art unsupervised deep hashing (DH) algorithms suffer from severe performance degradation for unsupervised scenarios. One of the main reasons is that the ad-hoc encoding process cannot properly capture the visual feature distribution. In this paper, we propose a novel unsupervised framework that has two main contributions: 1) we convert the unsupervised DH model into supervised by discovering pseudo labels; 2) the framework unifies likelihood maximization, mutual information maximization, and quantization error minimization so that the pseudo labels can maximumly preserve the distribution of visual features. Extensive experiments on three popular data sets demonstrate the advantages of the proposed method, which leads to significant performance improvement over the state-of-the-art unsupervised hashing algorithms.

Exploring Auxiliary Context: Discrete Semantic Transfer Hashing for Scalable Image Retrieval.

Unsupervised hashing can desirably support scalable content-based image retrieval for its appealing advantages of semantic label independence, memory, and search efficiency. However, the learned hash codes are embedded with limited discriminative semantics due to the intrinsic limitation of image representation. To address the problem, in this paper, we propose a novel hashing approach, dubbed as discrete semantic transfer hashing (DSTH). The key idea is to directly augment the semantics of discrete image hash codes by exploring auxiliary contextual modalities. To this end, a unified hashing framework is formulated to simultaneously preserve visual similarities of images and perform semantic transfer from contextual modalities. Furthermore, to guarantee direct semantic transfer and avoid information loss, we explicitly impose the discrete constraint, bit-uncorrelation constraint, and bit-balance constraint on hash codes. A novel and effective discrete optimization method based on augmented Lagrangian multiplier is developed to iteratively solve the optimization problem. The whole learning process has linear computation complexity and desirable scalability. Experiments on three benchmark data sets demonstrate the superiority of DSTH compared with several state-of-the-art approaches.

Metric learning via feature weighting for scalable image retrieval

Two dominant image retrieval schemes are based on local features indexed by an inverted index and global features indexed by compact hashing codes. They both demonstrate excellent scalability, but distinct strength for image retrieval. This motivates us to investigate how to fuse these two search schemes, to further enhance the retrieval effectiveness. Thus, we propose a novel metric learning method, namely Metric Learning via Feature Weighting (MLFW), to effectively fuse different features. MLFW learns the distance metric on individual feature as well as the weights of different features in a joint framework, to combine the learned distance obtained from all the individual feature and the early fusion. Furthermore, we design an efficient solution to optimize the objective function. Extensive experimental results conducted on real-life datasets show that the proposed MLFW outperforms the state-of-the-art methods in terms of search quality.

SiNC: Saliency-injected neural codes for representation and efficient retrieval of medical radiographs.

Medical image collections contain a wealth of information which can assist radiologists and medical experts in diagnosis and disease detection for making well-informed decisions. However, this objective can only be realized if efficient access is provided to semantically relevant cases from the ever-growing medical image repositories. In this paper, we present an efficient method for representing medical images by incorporating visual saliency and deep features obtained from a fine-tuned convolutional neural network (CNN) pre-trained on natural images. Saliency detector is employed to automatically identify regions of interest like tumors, fractures, and calcified spots in images prior to feature extraction. Neuronal activation features termed as neural codes from different CNN layers are comprehensively studied to identify most appropriate features for representing radiographs. This study revealed that neural codes from the last fully connected layer of the fine-tuned CNN are found to be the most suitable for representing medical images. The neural codes extracted from the entire image and salient part of the image are fused to obtain the saliency-injected neural codes (SiNC) descriptor which is used for indexing and retrieval. Finally, locality sensitive hashing techniques are applied on the SiNC descriptor to acquire short binary codes for allowing efficient retrieval in large scale image collections. Comprehensive experimental evaluations on the radiology images dataset reveal that the proposed framework achieves high retrieval accuracy and efficiency for scalable image retrieval applications and compares favorably with existing approaches.

Supervised graph hashing for histopathology image retrieval and classification.

In pathology image analysis, morphological characteristics of cells are critical to grade many diseases. With the development of cell detection and segmentation techniques, it is possible to extract cell-level information for further analysis in pathology images. However, it is challenging to conduct efficient analysis of cell-level information on a large-scale image dataset because each image usually contains hundreds or thousands of cells. In this paper, we propose a novel image retrieval based framework for large-scale pathology image analysis. For each image, we encode each cell into binary codes to generate image representation using a novel graph based hashing model and then conduct image retrieval by applying a group-to-group matching method to similarity measurement. In order to improve both computational efficiency and memory requirement, we further introduce matrix factorization into the hashing model for scalable image retrieval. The proposed framework is extensively validated with thousands of lung cancer images, and it achieves 97.98% classification accuracy and 97.50% retrieval precision with all cells of each query image used.

Quantization-based hashing: a general framework for scalable image and video retrieval

Nowadays, due to the exponential growth of user generated images and videos, there is an increasing interest in learning-based hashing methods. In computer vision, the hash functions are learned in such a way that the hash codes can preserve essential properties of the original space (or label information). Then the Hamming distance of the hash codes can approximate the data similarity. On the other hand, vector quantization methods quantize the data into different clusters based on the criteria of minimal quantization error, and then perform the search using look-up tables. While hashing methods using Hamming distance can achieve faster search speed, their accuracy is often outperformed by quantization methods with the same code length, due to the low quantization error and more flexible distance lookups. To improve the effectiveness of the hashing methods, in this work, we propose Quantization-based Hashing (QBH), a general framework which incorporates the advantages of quantization error reduction methods into conventional property preserving hashing methods. The learned hash codes simultaneously preserve the properties in the original space and reduce the quantization error, and thus can achieve better performance. Furthermore, the hash functions and a quantizer can be jointly learned and iteratively updated in a unified framework, which can be readily used to generate hash codes or quantize new data points. Importantly, QBH is a generic framework that can be integrated to different property preserving hashing methods and quantization strategies, and we apply QBH to both unsupervised and supervised hashing models as showcases in this paper. Experimental results on three large-scale unlabeled datasets (i.e., SIFT1M, GIST1M, and SIFT1B), three labeled datastes (i.e., ESPGAME, IAPRTC and MIRFLICKR) and one video dataset (UQ_VIDEO) demonstrate the superior performance of our QBH over existing unsupervised and supervised hashing methods.

Adaptive Hashing with Sparse Modification for Scalable Image Retrieval

Approximate Nearest Neighbor (ANN) search is a challenging problem with the explosive high-dimensional large-scale data in recent years. The promising technique for ANN search include hashing methods which generate compact binary codes by designing effective hash functions. However, lack of an optimal regularization is the key limitation of most of the existing hash functions. To this end, a new method called Adaptive Hashing with Sparse Modification (AHSM) is proposed. In AHSM, codes consist of vertices on the hypercube and the projection matrix is divided into two separate matrices. Data is rotated through a orthogonal matrix first and modified by a sparse matrix. Here the sparse matrix needs to be learned as a regularization item of hash function which is used to avoid overfitting and reduce quantization distortion. Totally, AHSM has two advantages: improvement of the accuracy without any time cost increasement. Furthermore, we extend AHSM to a supervised version, called Supervised Adaptive Hashing with Sparse Modification (SAHSM), by introducing Canonical Correlation Analysis (CCA) to the original data. Experiments show that the AHSM method stably surpasses several state-of-the-art hashing methods on four data sets. And at the same time, we compare three unsupervised hashing methods with their corresponding supervised version (including SAHSM) on three data sets with labels known. Similarly, SAHSM outperforms other methods on most of the hash bits.

Scalable Image Retrieval by Sparse Product Quantization

Fast Approximate Nearest Neighbor (ANN) search technique for high-dimensional feature indexing and retrieval is the crux of large-scale image retrieval. A recent promising technique is Product Quantization, which attempts to index high-dimensional image features by decomposing the feature space into a Cartesian product of low dimensional subspaces and quantizing each of them separately. Despite the promising results reported, their quantization approach follows the typical hard assignment of traditional quantization methods, which may result in large quantization errors and thus inferior search performance. Unlike the existing approaches, in this paper, we propose a novel approach called Sparse Product Quantization (SPQ) to encoding the high-dimensional feature vectors into sparse representation. We optimize the sparse representations of the feature vectors by minimizing their quantization errors, making the resulting representation is essentially close to the original data in practice. Experiments show that the proposed SPQ technique is not only able to compress data, but also an effective encoding technique. We obtain state-of-the-art results for ANN search on four public image datasets and the promising results of content-based image retrieval further validate the efficacy of our proposed method.

Learning to Hash With Optimized Anchor Embedding for Scalable Retrieval.

Sparse representation and image hashing are powerful tools for data representation and image retrieval respectively. The combinations of these two tools for scalable image retrieval, i.e., sparse hashing (SH) methods, have been proposed in recent years and the preliminary results are promising. The core of those methods is a scheme that can efficiently embed the (high-dimensional) image features into a low-dimensional Hamming space, while preserving the similarity between features. Existing SH methods mostly focus on finding better sparse representations of images in the hash space. We argue that the anchor set utilized in sparse representation is also crucial, which was unfortunately underestimated by the prior art. To this end, we propose a novel SH method that optimizes the integration of the anchors, such that the features can be better embedded and binarized, termed as Sparse Hashing with Optimized Anchor Embedding. The central idea is to push the anchors far from the axis while preserving their relative positions so as to generate similar hashcodes for neighboring features. We formulate this idea as an orthogonality constrained maximization problem and an efficient and novel optimization framework is systematically exploited. Extensive experiments on five benchmark image data sets demonstrate that our method outperforms several state-of-the-art related methods.

Learning Short Binary Codes for Large-scale Image Retrieval.

Large-scale visual information retrieval has become an active research area in this big data era. Recently, hashing/binary coding algorithms prove to be effective for scalable retrieval applications. Most existing hashing methods require relatively long binary codes (i.e., over hundreds of bits, sometimes even thousands of bits) to achieve reasonable retrieval accuracies. However, for some realistic and unique applications, such as on wearable or mobile devices, only short binary codes can be used for efficient image retrieval due to the limitation of computational resources or bandwidth on these devices. In this paper, we propose a novel unsupervised hashing approach called min-cost ranking (MCR) specifically for learning powerful short binary codes (i.e., usually the code length shorter than 100 b) for scalable image retrieval tasks. By exploring the discriminative ability of each dimension of data, MCR can generate one bit binary code for each dimension and simultaneously rank the discriminative separability of each bit according to the proposed cost function. Only top-ranked bits with minimum cost-values are then selected and grouped together to compose the final salient binary codes. Extensive experimental results on large-scale retrieval demonstrate that MCR can achieve comparative performance as the state-of-the-art hashing algorithms but with significantly shorter codes, leading to much faster large-scale retrieval.