Image Retrieval Tasks Research Articles

Enhanced bag of visual words representations for content based image retrieval: a comparative study

The exponential growth of digital image data poses numerous open problems to computer vision researchers. In this regard, designing an efficient and more accurate mechanism that finds and retrieve desired images from large repositories is of greater importance. To this end, various types of content based image retrieval (CBIR) systems have been developed. A typical CBIR system enables the search and retrieval of desired images from large databases that are similar to a given query image by means of automatically extracted visual features from image pixels. In CBIR domain, the bag of visual words (BoVW) model is one of the most widely used feature representation scheme and there exist a number of image retrieval frameworks based on BoVW model. It has been observed that most of them demonstrated promising results for the task of medium and large scale image retrieval. However, image retrieval literature lacks a comparative evaluation of these extended BoVW formulations. To this end, this paper aims to categorize and evaluate the existing BoVW model based formulations for the task of content based image retrieval. The commonly used datasets and the evaluation metrics to assess the retrieval effectiveness of these existing models are discussed. Moreover, quantitative evaluation of state of the art image retrieval systems based on BoVW model is also provided. Finally, certain promising directions for future research are proposed on the basis of the existing models and the demand from real-world.

Benchmarking unsupervised near-duplicate image detection

Unsupervised near-duplicate detection has many practical applications ranging from social media analysis and web-scale retrieval, to digital image forensics. It entails running a threshold-limited query on a set of descriptors extracted from the images, with the goal of identifying all possible near-duplicates, while limiting the false positives due to visually similar images. Since the rate of false alarms grows with the dataset size, a very high specificity is thus required, up to 1–10−9 for realistic use cases; this important requirement, however, is often overlooked in literature. In recent years, descriptors based on deep convolutional neural networks have matched or surpassed traditional feature extraction methods in content-based image retrieval tasks. To the best of our knowledge, ours is the first attempt to establish the performance range of deep learning-based descriptors for unsupervised near-duplicate detection on a range of datasets, encompassing a broad spectrum of near-duplicate definitions. We leverage both established and new benchmarks, such as the Mir-Flick Near-Duplicate (MFND) dataset, in which a known ground truth is provided for all possible pairs over a general, large scale image collection. To compare the specificity of different descriptors, we reduce the problem of unsupervised detection to that of binary classification of near-duplicate vs. not-near-duplicate images. The latter can be conveniently characterized using Receiver Operating Curve (ROC). Our findings in general favor the choice of fine-tuning deep convolutional networks, as opposed to using off-the-shelf features, but differences at high specificity settings depend on the dataset and are often small. The best performance was observed on the MFND benchmark, achieving 96% sensitivity at a false positive rate of 1.43×10−6.

Deep embedding learning with adaptive large margin N-pair loss for image retrieval and clustering

Deep embedding learning becomes more attractive for discriminative feature learning, but many methods still require hard-class mining, which is computationally complex and performance-sensitive. To this end, Adaptive Large Margin N-Pair loss (ALMN) is proposed to address the aforementioned issues. First, the class center is adopted as the anchor point to avoid the difficulty on anchor selection. Then instead of exploring hard example-mining strategy, we introduce the adaptive large margin constraint, where a novel geometrical Virtual Point Generating (VPG) method is proposed to convert a fixed margin into a local-adaptive angular margin, by automatically generating a boundary training sample in feature space. The effectiveness of our method is demonstrated on fine-grained image retrieval and clustering tasks using six popular databases, including CUB, CARS, Flowers, Aircraft, Stanford Online Products and In-Shop Clothes. The results show that the proposed method achieves better performance than other state-of-the-art methods, such as N-Pair loss, Lifted loss and Triplet loss.

Multi-level magnification correlation hashing for scalable histopathological image retrieval

Content-based Medical Image Retrieval (CBMIR) presents promising results in computer-aided diagnosis which provides the clinicians with interpretative guidance based on the visual similarity. In this paper, we focus on the tasks of histopathological image retrieval for breast cancer diagnosis. Instead of the holistic-based and cell-based methods proposed in previous works, we propose a multi-magnification correlation hashing (MMCH) framework that learns the discriminative binary codes for histopathological images. The patch-link graph is constructed with both low and high magnification images where the semantic links are propagated from the pre-labeled patches to unlabeled data. We then learn the binary hashing codes based on the global labels from the low-magnification images with structure-regularization. The experiments on BreakHis dataset and PLOSONE dataset demonstrate that MMCH outperforms the previous hashing methods on histopathological image retrieval.

Attribute-Guided Network for Cross-Modal Zero-Shot Hashing.

Zero-shot hashing (ZSH) aims at learning a hashing model that is trained only by instances from seen categories but can generate well to those of unseen categories. Typically, it is achieved by utilizing a semantic embedding space to transfer knowledge from seen domain to unseen domain. Existing efforts mainly focus on single-modal retrieval task, especially image-based image retrieval (IBIR). However, as a highlighted research topic in the field of hashing, cross-modal retrieval is more common in real-world applications. To address the cross-modal ZSH (CMZSH) retrieval task, we propose a novel attribute-guided network (AgNet), which can perform not only IBIR but also text-based image retrieval (TBIR). In particular, AgNet aligns different modal data into a semantically rich attribute space, which bridges the gap caused by modality heterogeneity and zero-shot setting. We also design an effective strategy that exploits the attribute to guide the generation of hash codes for image and text within the same network. Extensive experimental results on three benchmark data sets (AwA, SUN, and ImageNet) demonstrate the superiority of AgNet on both cross-modal and single-modal zero-shot image retrieval tasks.

Montage based 3D Medical Image Retrieval from Traumatic Brain Injury Cohort using Deep Convolutional Neural Network.

Brain imaging analysis on clinically acquired computed tomography (CT) is essential for the diagnosis, risk prediction of progression, and treatment of the structural phenotypes of traumatic brain injury (TBI). However, in real clinical imaging scenarios, entire body CT images (e.g., neck, abdomen, chest, pelvis) are typically captured along with whole brain CT scans. For instance, in a typical sample of clinical TBI imaging cohort, only ~15% of CT scans actually contain whole brain CT images suitable for volumetric brain analyses; the remaining are partial brain or non-brain images. Therefore, a manual image retrieval process is typically required to isolate the whole brain CT scans from the entire cohort. However, the manual image retrieval is time and resource consuming and even more difficult for the larger cohorts. To alleviate the manual efforts, in this paper we propose an automated 3D medical image retrieval pipeline, called deep montage-based image retrieval (dMIR), which performs classification on 2D montage images via a deep convolutional neural network. The novelty of the proposed method for image processing is to characterize the medical image retrieval task based on the montage images. In a cohort of 2000 clinically acquired TBI scans, 794 scans were used as training data, 206 scans were used as validation data, and the remaining 1000 scans were used as testing data. The proposed achieved accuracy=1.0, recall=1.0, precision=1.0, f1=1.0 for validation data, while achieved accuracy=0.988, recall=0.962, precision=0.962, f1=0.962 for testing data. Thus, the proposed dMIR is able to perform accurate CT whole brain image retrieval from large-scale clinical cohorts.

Grouped attribute strength-based image retrieval

Visual attributes, as a bridge between humans and machines, have played an important role in various applications. We focus on attribute-based image retrieval. Unlike early works that either initiate the image search via binary attributes or refine the search results via relative attributes, we propose to realize image retrieval using grouped attribute strength, which are learned from both binary attributes and relative attributes. First, early methods are employed to predict the binary attributes and sort the relative attributes in descending order in different situations. Second, we group the sorted images corresponding to each attribute and combine the learned binary and relative attributes to learn the grouped attribute strength of the images, which result in a new attribute feature vector for each image. Finally, we apply the learned grouped attribute strength to different retrieval models to realize our image retrieval tasks. We demonstrate the approach on LFW-10 and Shoes datasets and show its clear advantages over traditional binary attribute-based retrieval methods.

Applying uncertain frequent pattern mining to improve ranking of retrieved images

Immense increase in digital images demands an effective image retrieval system. In text based image retrieval systems, textual keywords are used for indexing. The query keywords are matched with the keywords associated with images to perform image retrieval task. Users usually demand higher proportion of the query keywords in the retrieved images than the other undesired keywords. Existing image retrieval systems retrieve images that do not contain the query keywords either in equal or higher proportion than other keywords. This paper proposes a new image retrieval system that applies fuzzy ontology and uncertain frequent pattern mining for image retrieval to resolve this issue. Image content is represented in terms of concepts and categories. Fuzzy ontology is constructed by utilizing the concepts and categories associated with the images. Uncertain frequent pattern mining is then applied on the association that exists among the concepts in images. These patterns assist in retrieving images, which contain the required query keywords in high proportion than other keywords. The ranking of retrieved images is evaluated with two different measures, i.e., mean and variance and difference at various thresholds of concepts’ occurrences in images. Experimental results show that the proposed image retrieval system performs better than existing ontology based retrieval systems.

Semi-supervised and active learning through Manifold Reciprocal kNN Graph for image retrieval

A massive and ever growing amount of data collections, including visual and multimedia content are available today. Such content usually possesses additional information, as text or other metadata, to form a rather sparse and noisy, yet rich and diverse source of annotation. Although the text-based retrieval models are well established, they ignore the rich source of information encoded in the visual data. In contrast, the promising content-based retrieval technologies, capable of considering the multimedia content, still face obstacles for mapping the low level features into high level semantic concepts. Supervised approaches based on relevance feedback techniques have been employed for mitigating such gap on visual retrieval tasks. Although often quite effective, such methods rely only on labeled data, which can severely impact the retrieval effectiveness when the number of user interventions is insufficient. In this scenario, the retrieval approaches are ideally suitable for the emerging weakly supervised and active learning technology to semi-autonomously explore data collections by taking into account the relationships among multimedia objects and saving the user’s efforts. In this paper, we discuss a novel semi-supervised learning algorithm for image retrieval tasks. While a manifold learning algorithm uses a reciprocal kNN graph to analyze the unlabeled data, the labeled information obtained through user interactions are represented using similarity sets. Both labeled and unlabeled information are modelled in terms of ranking information to allow a strict link between them. Experimental results obtained on various public datasets and several different visual features have demonstrated the effectiveness of the proposed approach.

OCEAN: Object-centric arranging network for self-supervised visual representations learning

Learning visual representations plays an important role in computer vision and machine learning applications. It facilitates a model to understand and perform high-level tasks intelligently. A common approach for learning visual representations is supervised one which requires a huge amount of human annotations to train the model. This paper presents a self-supervised approach which learns visual representations from input images without human annotations. We learn the correct arrangement of object proposals to represent an image using a convolutional neural network (CNN) without any manual annotations. We hypothesize that the network trained for solving this problem requires the embedding of semantic visual representations. Unlike existing approaches that use uniformly sampled patches, we relate object proposals that contain prominent objects and object parts. More specifically, we discover the representation that considers overlap, inclusion, and exclusion relationship of proposals as well as their relative position. This allows focusing on potential objects and parts rather than on clutter. We demonstrate that our model outperforms existing self-supervised learning methods and can be used as a generic feature extractor by applying it to object detection, classification, action recognition, image retrieval, and semantic matching tasks.

Learning non-metric visual similarity for image retrieval

Measuring visual similarity between two or more instances within a data distribution is a fundamental task in image retrieval. Theoretically, non-metric distances are able to generate a more complex and accurate similarity model than metric distances, provided that the non-linear data distribution is precisely captured by the system. In this work, we explore neural networks models for learning a non-metric similarity function for instance search. We argue that non-metric similarity functions based on neural networks can build a better model of human visual perception than standard metric distances. As our proposed similarity function is differentiable, we explore a real end-to-end trainable approach for image retrieval, i.e. we learn the weights from the input image pixels to the final similarity score. Experimental evaluation shows that non-metric similarity networks are able to learn visual similarities between images and improve performance on top of state-of-the-art image representations, boosting results in standard image retrieval datasets with respect standard metric distances.

Composite description based on Salient Contours and Color information for CBIR tasks.

This paper introduces a novel image descriptor for content based image retrieval tasks that integrates contour and color information into a compact vector. Loosely inspired by the human visual system and its mechanisms in efficiently identifying visual saliency, operations are performed on a fixed lattice of discrete positions by a set of edge detecting kernels that calculate region derivatives at different scales and orientation. The description method utilizes a weighted edge histogram where bins are populated on the premise of whether the regions contain edges belonging to the salient contours, while the discriminative power is further enhanced by integrating regional quantized color information. The proposed technique is both efficient and adaptive to the specifics of each depiction, while it does not need any training data to adjust parameters. Experimental evaluation conducted on seven benchmarking datasets against 13 well known global descriptors along with SIFT, SURF implementations (both in VLAD and BOVW), highlight the effectiveness and efficiency of the proposed descriptor.

Retrieving color patterns on surface meshes using edgeLBP descriptors

Texture image retrieval and classification tasks are conducted widely in computer vision but few studies have considered the same problem for decorations and textures applied to surfaces. Thus, in order to address this problem, we propose an algorithm for retrieving and classifying color patterns over surface meshes. In this study, we explored the capability of edgeLBP descriptors for retrieving and classifying color patterns over surface tessellations. Our experimental results demonstrated that these descriptors performed well at pattern retrieval, including for patterns on degraded and corrupted archeological fragments. We compared the performance of the edgeLBP descriptors with a recent benchmark and conducted an extended analysis of the robustness of this technique to mesh remeshing and surface decimation.

Integration of global and local features based on hybrid similarity matching scheme for medical image retrieval system

Similarity measure is a challenging task in Content-Based Medical Image Retrieval (CBMIR) systems and the matching scheme is designed to improve the retrieval performance. However, there are several major shortcomings with conventional approaches for a matching scheme which can extensively affect their application of Medical Image Retrieval (MIR). To overcome the issues, in this paper a Multi-Level Matching (MLM) method for MIR using hybrid feature similarity is proposed. Here, images are represented by multi-level features including local level and global level. The Colour and Edge Directivity Descriptor (CEDD) is used as a colour and edge-based descriptor. Speeded-Up Robust Features (SURF) and Local Binary Pattern (LBP) are used as a local descriptor. The hybrid of both global and local features yields enhanced retrieval accuracy, which is analysed over collected image databases. From the experiment, the proposed method achieves better accuracy value about 92%, which is higher than other methods.

Medical Image Retrieval Based on Convolutional Neural Network and Supervised Hashing

In recent years, with extensive application in image retrieval and other tasks, a convolutional neural network (CNN) has achieved outstanding performance. In this paper, a new content-based medical image retrieval (CBMIR) framework using CNN and hash coding is proposed. The new framework adopts a Siamese network in which pairs of images are used as inputs, and a model is learned to make images belonging to the same class have similar features by using weight sharing and a contrastive loss function. In each branch of the network, CNN is adapted to extract features, followed by hash mapping, which is used to reduce the dimensionality of feature vectors. In the training process, a new loss function is designed to make the feature vectors more distinguishable, and a regularization term is added to encourage the real value outputs to approximate the desired binary values. In the retrieval phase, the compact binary hash code of the query image is achieved from the trained network and is subsequently compared with the hash codes of the database images. We experimented on two medical image datasets: the cancer imaging archive-computed tomography (TCIA-CT) and the vision and image analysis group/international early lung cancer action program (VIA/I-ELCAP). The results indicate that our method is superior to existing hash methods and CNN methods. Compared with the traditional hashing method, feature extraction based on CNN has advantages. The proposed algorithm combining a Siamese network with the hash method is superior to the classical CNN-based methods. The application of a new loss function can effectively improve retrieval accuracy.

Balanced Deep Supervised Hashing

Recently, Convolutional Neural Network (CNN) based hashing method has achieved its promising performance for image retrieval task. However, tackling the discrepancy between quantization error minimization and discriminability maximization of network outputs simultaneously still remains unsolved. Motivated by the concern, we propose a novel Balanced Deep Supervised Hashing (BDSH) based on variant posterior probability to learn compact discriminability-preserving binary code for large scale image data. Distinguished from the previous works, BDSH can search an equilibrium point within the discrepancy. Towards the goal, a delicate objective function is utilized to maximize the discriminability of the output space with the variant posterior probability of the pair-wise label. A quantization regularizer is utilized as a relaxation from real-value outputs to the desired discrete values (e.g., -1/+1). Extensive experiments on the benchmark datasets show that our method can yield state-of-the-art image retrieval performance from various perspectives.

Random linear interpolation data augmentation for person re-identification

Person Re-Identification (person re-ID) is an image retrieval task which identifies the same person in different camera views. Generally, a good person re-ID model requires a large dataset containing over 100000 images to reduce the risk of over-fitting. Most current handcrafted person re-ID datasets, however, are insufficient for training a learning model with high generalization ability. In addition, the lacking of images with various levels of occlusion is still remaining in most existing datasets. Motivated by these two problems, this paper proposes a new data augmentation method called Random Linear Interpolation that can enlarge the sizes of person re-ID datasets and improve the generalization ability of the learning model. The key enabler of our approach is generating fused images by interpolating pairs of original images. In other words, the innovation of the proposed approach is considering data augmentation between two random samples. Plenty of experimental results demonstrates that the proposed method is effective to improve baseline models. On Market1501 and DukeMTMC-reID datasets, our approach can achieve 92.71% and 82.19% rank-1 accuracy, respectively.

Unsupervised semantic-based convolutional features aggregation for image retrieval

Deep features extracted from the convolutional layers of pre-trained CNNs have been widely used in the image retrieval task. These features, however, are in a large number and probably cannot be directly used for similarity evaluation due to lack of efficiency. Thus, it is of great importance to study how to aggregate deep features into a global yet distinctive image vector. This paper first introduces a simple but effective method to select informative features based on semantic content of feature maps. Then, we propose an effective channel weighting method (CW) for selected features by analyzing relations between the discriminative activation and distribution parameters of feature maps, including standard variance, non-zero responses and sum value. Furthermore, we provide a solution to pick semantic detectors that are independent on gallery images. Based on the aforementioned three strategies, we derive a global image vector generation method, and demonstrate its state-of-the-art performance on benchmark datasets.

Densely-Connected Multi-Magnification Hashing for Histopathological Image Retrieval.

Content-based medical image retrieval is an important computer-aided diagnosis technique providing the clinicians with interpretative references based on visual similarity. In this paper, we focus on the tasks of histopathological image retrieval for breast cancer diagnosis. The densely-connected multi-magnification (DCMMH) framework is proposed to generate the discriminative binary codes by exploiting the histopathological images with multiple magnification factors. The low-magnification images are boosted by the accumulated similarity based on local patches that also regularize the feature learning of high-magnification images. In order to fully utilize the information across different magnification levels, a densely-connected architecture is finally deployed for high-low magnification pairs of datasets. Experiments on BreakHis dataset demonstrate that, DCMMH outperforms the previous hashing methods on histopathological image retrieval.

Are we done with object recognition? The iCub robot’s perspective

We report on an extensive study of the benefits and limitations of current deep learning approaches to object recognition in robot vision scenarios, introducing a novel dataset used for our investigation. To avoid the biases in currently available datasets, we consider a natural human–robot interaction setting to design a data-acquisition protocol for visual object recognition on the iCub humanoid robot. Analyzing the performance of off-the-shelf models trained off-line on large-scale image retrieval datasets, we show the necessity for knowledge transfer. We evaluate different ways in which this last step can be done, and identify the major bottlenecks affecting robotic scenarios. By studying both object categorization and identification problems, we highlight key differences between object recognition in robotics applications and in image retrieval tasks, for which the considered deep learning approaches have been originally designed. In a nutshell, our results confirm the remarkable improvements yield by deep learning in this setting, while pointing to specific open challenges that need be addressed for seamless deployment in robotics.