Content-based Image Retrieval Tasks Research Articles

Benchmarking Deep Learning-Based Image Retrieval of Oral Tumor Histology.

Oral tumors necessitate a dependable computer-assisted pathological diagnosis system considering their rarity and diversity. A content-based image retrieval (CBIR) system using deep neural networks has been successfully devised for digital pathology. No CBIR system for oral pathology has been investigated because of the lack of an extensive image database and feature extractors tailored to oral pathology. This study uses a large CBIR database constructed from 30 categories of oral tumors to compare deep learning methods as feature extractors. The highest average area under the receiver operating characteristic curve (AUC) was achieved by models trained on database images using self-supervised learning (SSL) methods (0.900 with SimCLR and 0.897 with TiCo). The generalizability of the models was validated using query images from the same cases taken with smartphones. When smartphone images were tested as queries, both models yielded the highest mean AUC (0.871 with SimCLR and 0.857 with TiCo). We ensured the retrieved image result would be easily observed by evaluating the top 10 mean accuracies and checking for an exact diagnostic category and its differential diagnostic categories. Training deep learning models with SSL methods using image data specific to the target site is beneficial for CBIR tasks in oral tumor histology to obtain histologically meaningful results and high performance. This result provides insight into the effective development of a CBIR system to help improve the accuracy and speed of histopathology diagnosis and advance oral tumor research in the future.

Highly compressed image representation for classification and content retrieval

In this paper, we propose a new method of representing images using highly compressed features for classification and image content retrieval – called PCA-ResFeats. They are obtained by fusing high- and low-level features from the outputs of ResNet-50 residual blocks and applying to them principal component analysis, which leads to a significant reduction in dimensionality. Further on, by applying a floating-point compression, we are able to reduce the memory required to store a single image by up to 1,200 times compared to jpg images and 220 times compared to features obtained by simple output fusion of ResNet-50. As a result, the representation of a single image from the dataset can be as low as 35 bytes on average. In comparison with the classification results on features from fusion of the last ResNet-50 residual block, we achieve a comparable accuracy (no worse than five percentage points), while preserving two orders of magnitude data compression. We also tested our method in the content-based image retrieval task, achieving better results than other known methods using sparse features. Moreover, our method enables the creation of concise summaries of image content, which can find numerous applications in databases.

Medical image retrieval via nearest neighbor search on pre-trained image features

Nearest neighbor search, also known as NNS, is a technique used to locate the points in a high-dimensional space closest to a given query point. This technique has multiple applications in medicine, such as searching large medical imaging databases, disease classification, and diagnosis. However, when the number of points is significantly large, the brute-force approach for finding the nearest neighbor becomes computationally infeasible. Therefore, various approaches have been developed to make the search faster and more efficient to support the applications. With a focus on medical imaging, this paper proposes DenseLinkSearch (DLS), an effective and efficient algorithm that searches and retrieves the relevant images from heterogeneous sources of medical images. Towards this, given a medical database, the proposed algorithm builds an index that consists of pre-computed links of each point in the database. The search algorithm utilizes the index to efficiently traverse the database in search of the nearest neighbor. We also explore the role of medical image feature representation in content-based medical image retrieval tasks. We propose a Transformer-based feature representation technique that outperformed the existing pre-trained Transformer-based approaches on benchmark medical image retrieval datasets. We extensively tested the proposed NNS approach and compared the performance with state-of-the-art NNS approaches on benchmark datasets and our created medical image datasets. The proposed approach outperformed the existing approaches in terms of retrieving accurate neighbors and retrieval speed. In comparison to the existing approximate NNS approaches, our proposed DLS approach outperformed them in terms of lower average time per query and ≥99% R@10 on 11 out of 13 benchmark datasets. We also found that the proposed medical feature representation approach is better for representing medical images compared to the existing pre-trained image models. The proposed feature extraction strategy obtained an improvement of 9.37%, 7.0%, and 13.33% in terms of P@5, P@10, and P@20, respectively, in comparison to the best-performing pre-trained image model. The source code and datasets of our experiments are available at https://github.com/deepaknlp/DLS.

Co-attention enabled content-based image retrieval

Content-based image retrieval (CBIR) aims to provide the most similar images to a given query. Feature extraction plays an essential role in retrieval performance within a CBIR pipeline. Current CBIR studies would either uniformly extract feature information from the input image and use it directly or employ some trainable spatial weighting module which is then used for similarity comparison between pairs of query and candidate matching images. These spatial weighting modules are normally query non-sensitive and only based on the knowledge learned during the training stage. They may focus towards incorrect regions, especially when the target image is not salient or is surrounded by distractors. This paper proposes an efficient query sensitive co-attention11“Co-attention” in this paper refers to spatial attention conditioned on the query content. mechanism for large-scale CBIR tasks. In order to reduce the extra computation cost required by the query sensitivity to the co-attention mechanism, the proposed method employs clustering of the selected local features. Experimental results indicate that the co-attention maps can provide the best retrieval results on benchmark datasets under challenging situations, such as having completely different image acquisition conditions between the query and its match image.

A New Approach to Descriptors Generation for Image Retrieval by Analyzing Activations of Deep Neural Network Layers.

In this brief, we consider the problem of descriptors construction for the task of content-based image retrieval using deep neural networks. The idea of neural codes, based on fully connected layers' activations, is extended by incorporating the information contained in convolutional layers. It is known that the total number of neurons in the convolutional part of the network is large and the majority of them have little influence on the final classification decision. Therefore, in this brief, we propose a novel algorithm that allows us to extract the most significant neuron activations and utilize this information to construct effective descriptors. The descriptors consisting of values taken from both the fully connected and convolutional layers perfectly represent the whole image content. The images retrieved using these descriptors match semantically very well to the query image, and also, they are similar in other secondary image characteristics, such as background, textures, or color distribution. These features of the proposed descriptors are verified experimentally based on the IMAGENET1M dataset using the VGG16 neural network. For comparison, we also test the proposed approach on the ResNet50 network.

Image Retrieval Algorithm Based on Locality-Sensitive Hash Using Convolutional Neural Network and Attention Mechanism

With the continuous progress of image retrieval technology, in the field of image retrieval, the speed of a search for a desired image from a great deal of image data becomes a hot issue. Convolutional Neural Networks (CNN) have been used in the field of image retrieval. However, many image retrieval systems based on CNN have a poor ability to express image features, resulting in a series of problems such as low retrieval accuracy and robustness. When the target image is retrieved from a large amount of image data, the vector dimension after image coding is high and the retrieval efficiency is low. Locality-sensitive hash is a method to find similar data from massive high latitude data. It reduces the data dimension of the original spatial data through hash coding and conversion, and can also maintain the similarity between the data. The retrieval time and space complexity are low. Therefore, this paper proposes a locality-sensitive hash image retrieval method based on CNN and the attention mechanism. The steps of the method are as follows: using the ResNet50 network as the feature extractor of the image, adding the attention module after the convolution layer of the model, and using the output of the network full connection layer to retrieve the features of the image database, then using the local-sensitive hash algorithm to hash code the image features of the database to reduce the dimension and establish the index, and finally measuring the features of the image to be retrieved and the image database to get the most similar image, completing the content-based image retrieval task. The method in this paper is compared with other image retrieval methods on corel1k and corel5k datasets. The experimental results show that this method can effectively improve the accuracy of image retrieval, and the retrieval efficiency is significantly improved. It also has higher robustness in different scenarios.

J-EDA: A workbench for tuning similarity and diversity search parameters in content-based image retrieval

Similarity searches can be modeled by means of distances following the Metric Spaces Theory and constitute a fast and explainable query mechanism behind content-based image retrieval (CBIR) tasks. However, classical distance-based queries, e.g., Range and k-Nearest Neighbors, may be unsuitable for exploring large datasets because the retrieved elements are often similar among themselves. Although similarity searching is enriched with the imposition of rules to foster result diversification, the fine-tuning of the diversity query is still an open issue, which is is usually carried out with and a non-optimal expensive computational inspection. This paper introduces J-EDA, a practical workbench implemented in Java that supports the tuning of similarity and diversity search parameters by enabling the automatic and parallel exploration of multiple search settings regarding a user-posed content-based image retrieval task. J-EDA implements a wide variety of classical and diversity-driven search queries, as well as many CBIR settings such as feature extractors for images, distance functions, and relevance feedback techniques. Accordingly, users can define multiple query settings and inspect their performances for spotting the most suitable parameterization for a content-based image retrieval problem at hand. The workbench reports the experimental performances with several internal and external evaluation metrics such as P × R and Mean Average Precision (mAP), which are calculated towards either incremental or batch procedures performed with or without human interaction.

Content Based Image Retrieval Using Deep Learning Convolutional Neural Network

Content-based image retrieval (CBIR) is a widely used method for image retrieval from large and unlabeled image collections. However, users are not satisfied with the traditional methods of retrieving information. Moreover the abundance of online networks for production and distribution, as well as the quantity of images accessible to consumers, continues to expand. Therefore, in many areas, permanent as well as widespread digital image processing takes place. Therefore, the rapid access to these large image databases as well as the extraction of identical images from this large set of images from a given image (Query) pose significant challenges as well as involves efficient techniques. A CBIR system’s efficiency depends fundamentally on the calculation of feature representation as well as similarity. For this purpose, they present a basic but powerful deep learning system focused on Convolutional Neural Networks (CNN) and composed of feature extraction and classification for fast image retrieval. We get some promising findings from many detailed observational studies for a number of CBIR tasks using image database, which reveals some valuable lessons for improving the efficiency of CBIR. CBIR systems allow another image dataset to locate related images to such a query image. The search per picture function of Google search has to be the most popular CBIR method.

A genetic algorithm approach for image representation learning through color quantization

Over the last decades, hand-crafted feature extractors have been used to encode image visual properties into feature vectors. Recently, data-driven feature learning approaches have been successfully explored as alternatives for producing more representative visual features. In this work, we combine both research venues, focusing on the color quantization problem. We propose two data-driven approaches to learn image representations through the search for optimized quantization schemes, which lead to more effective feature extraction algorithms and compact representations. Our strategy employs Genetic Algorithm, a soft-computing apparatus successfully utilized in Information-retrieval-related optimization problems. We hypothesize that changing the quantization affects the quality of image description approaches, leading to effective and efficient representations. We evaluate our approaches in content-based image retrieval tasks, considering eight well-known datasets with different visual properties. Results indicate that the approach focused on representation effectiveness outperformed baselines in all tested scenarios. The other approach, which also considers the size of created representations, produced competitive results keeping or even reducing the dimensionality of feature vectors up to 25%.

Convolutional neural network based dictionary learning to create hash codes for content-based image retrieval

Abstract This study investigates the suitability of sparse vectors in the dictionary learning (DL) method for content-based image retrieval (CBIR) tasks. Since DL usually performs the learning process in an unsupervised manner, it cannot generate robust features for the retrieval task, especially if a complex background is involved. In order to overcome this drawback, a DL approach using the feature representation power of the convolutional neural network (CNN) is proposed. The initialization values of the dictionaries in the proposed CNN based DL method are taken randomly from the middle layers of the CNN architecture. The vector of each image obtained from the CNN architecture is used as DL input. The lambda vectors produced by the DL structure are converted into binaries. In this way, DL acts as a hash code generator. The performance of the proposed framework is tested on modified COREL dataset. The results prove that it is an open-to-improvement approach and is promising.

Study of Content Based Image Retrieval Using Data Mining Techniques

<p>The field of image retrieval has been an active research area for several decades and has been paid more and more attention in recent years as a result of the dramatic and fast increase in the volume of digital images. Content-based image retrieval (CBIR) is a new but widely adopted method for finding images from vast and un annotated image databases. In recent years, a variety of techniques have been developed to improve the performance of CBIR. In reaction to the needs of users, who feel problems connected with traditional methods of image searching and indexing, researchers focus their interest on techniques for retrieving images on the basis of automatically-derived features, often denoted as Content-Based Image Retrieval (CBIR). CBIR systems index the media documents using salient features extracted from the actual media rather than by textual annotations. Query by content is nowadays a very active research field, with many systems being developed by industrial and academic teams. Results performed by these teams are really promising. The situation gets diametrically different when we move our attention from the usual CBIR task, i.e. the retrieval of images which are similar (as a whole) to the query image, to the task “find all images that contain the query image”. The proposed CBIR technique uses more than one clustering techniques to improve the performance of CBIR. This optimized method makes use of K-means and Hierarchical clustering technique to improve the execution time and performance of image retrieval systems in high dimensional sets. In this similarity measure is totally based on colors. In this paper more focus area is the way of combination of clustering technique in order to get faster output of images. In this paper the clustering techniques are discussed and analyzed. Also, we propose a method HDK that uses more than one clustering technique to improve the performance of CBIR. This method makes use of hierarchical and divides and conquers K-means clustering technique with equivalency and compatible relation concepts to improve the performance of the K-Means for using in high dimensional datasets. It also introduced the feature like color, texture and shape for accurate and effective retrieval system.</p>

Fine-Grained Image Retrieval via Piecewise Cross Entropy loss

Fine-Grained Image Retrieval is an important problem in computer vision. It is more challenging than the task of content-based image retrieval because it has small diversity within the different classes but large diversity in the same class. Recently, the cross entropy loss can be utilized to make Convolutional Neural Network (CNN) generate distinguish feature for Fine-Grained Image Retrieval, and it can obtain further improvement with some extra operations, such as Normalize-Scale layer. In this paper, we propose a variant of the cross entropy loss, named Piecewise Cross Entropy loss function, for enhancing model generalization and promoting the retrieval performance. Besides, the Piecewise Cross Entropy loss is easy to implement. We evaluate the performance of the proposed scheme on two standard fine-grained retrieval benchmarks, and obtain significant improvements over the state-of-the-art, with 11.8% and 3.3% over the previous work on CARS196 and CUB-200-2011, respectively.

Benchmarking unsupervised near-duplicate image detection

Unsupervised near-duplicate detection has many practical applications ranging from social media analysis and web-scale retrieval, to digital image forensics. It entails running a threshold-limited query on a set of descriptors extracted from the images, with the goal of identifying all possible near-duplicates, while limiting the false positives due to visually similar images. Since the rate of false alarms grows with the dataset size, a very high specificity is thus required, up to 1–10−9 for realistic use cases; this important requirement, however, is often overlooked in literature. In recent years, descriptors based on deep convolutional neural networks have matched or surpassed traditional feature extraction methods in content-based image retrieval tasks. To the best of our knowledge, ours is the first attempt to establish the performance range of deep learning-based descriptors for unsupervised near-duplicate detection on a range of datasets, encompassing a broad spectrum of near-duplicate definitions. We leverage both established and new benchmarks, such as the Mir-Flick Near-Duplicate (MFND) dataset, in which a known ground truth is provided for all possible pairs over a general, large scale image collection. To compare the specificity of different descriptors, we reduce the problem of unsupervised detection to that of binary classification of near-duplicate vs. not-near-duplicate images. The latter can be conveniently characterized using Receiver Operating Curve (ROC). Our findings in general favor the choice of fine-tuning deep convolutional networks, as opposed to using off-the-shelf features, but differences at high specificity settings depend on the dataset and are often small. The best performance was observed on the MFND benchmark, achieving 96% sensitivity at a false positive rate of 1.43×10−6.

Saliency Inside: Learning Attentive CNNs for Content-based Image Retrieval.

In content-based image retrieval (CBIR), one of the most challenging and ambiguous tasks are to correctly understand the human query intention and measure its semantic relevance with images in the database. Due to the impressive capability of visual saliency in predicting human visual attention that is closely related to the query intention, this paper attempts to explicitly discover the essential effect of visual saliency in CBIR via qualitative and quantitative experiments. Toward this end, we first generate the fixation density maps of images from a widely used CBIR dataset by using an eye-tracking apparatus. These ground-truth saliency maps are then used to measure the influence of visual saliency to the task of CBIR by exploring several probable ways of incorporating such saliency cues into the retrieval process. We find that visual saliency is indeed beneficial to the CBIR task, and the best saliency involving scheme is possibly different for different image retrieval models. Inspired by the findings, this paper presents two-stream attentive CNNs with saliency embedded inside for CBIR. The proposed network has two streams that simultaneously handle two tasks. The main stream focuses on extracting discriminative visual features that are tightly related to semantic attributes. Meanwhile, the auxiliary stream aims to facilitate the main stream by redirecting the feature extraction to the salient image content that human may pay attention to. By fusing these two streams into the Main and Auxiliary CNNs (MAC), image similarity can be computed as the human being does by reserving conspicuous content and suppressing irrelevant regions. Extensive experiments show that the proposed model achieves impressive performance in image retrieval on four public datasets.

Siamese graph convolutional network for content based remote sensing image retrieval

This paper deals with the problem of content-based image retrieval (CBIR) of very high resolution (VHR) remote sensing (RS) images using the notion of a novel Siamese graph convolution network (SGCN). The GCN model has recently gained popularity in learning representations for irregular domain data including graphs. In the same line, we argue the effectiveness of region adjacency graph (RAG) based image representations for VHR RS scenes in terms of localized regions. This technique captures important scene information which can further aid in a better image to image correspondence. However, standard GCN features, in general, lacks discriminative property for fine-grained classes. These features may not be optimal for the task of CBIR in many cases with coherent local characteristics. As a remedy, we propose the SGCN architecture for assessing the similarity between a pair of graphs which can be trained with the contrastive loss function. Given the RAG representations, the aim is to learn an embedding space that pulls semantically coherent images closer while pushing dissimilar samples far apart. In order to ensure a quick response while performing the retrieval using a given similarity measure, the embedding space is kept constrained. We implement the proposed embeddings for the task of CBIR for RS data on the popular UC-Merced dataset and the PatternNet dataset where improved performance can be observed.

Content-Based Image Retrieval : A Comprehensive Study

Learning efficient options illustrations and equivalency metric measures are imperative to the searching performance of a content-based image retrieval (CBIR) machine. Despite in depth analysis efforts for many years, it remains one amongst the foremost difficult open issues that significantly hinders the success of real- world CBIR systems. The key issue has been associated to the commonly known “linguistic gap” problem that exists between low-level image pixels captured by machines and high-level linguistics ideas perceived by humans. Among varied techniques, machine learning has been actively investigated as a potential direction to bridge the linguistics gap in the long run. Motivated by recent success of deep learning techniques for computer vision and other applications, In this paper, we'll conceive to address an open problem: if deep learning could be a hope for bridging the linguistics gap in CBIR and the way a lot of enhancements in CBIR tasks may be achieved by exploring the progressive deep learning methodologies for learning options illustrations and equivalency measures. Speci?cally, we'll investigate a framework of deep learning with application to CBIR tasks with an extensive set of empirical studies by examining a progressive deep learning technique (Convolutional Neural Networks) for CBIR tasks in varied settings. From our empirical studies, we found some encouraging results and summarized some vital insights for future analysis. CBIR tasks may be achieved by exploring the progressive deep learning techniques for learning options illustrations and equivalency measures.

Composite description based on Salient Contours and Color information for CBIR tasks.

This paper introduces a novel image descriptor for content based image retrieval tasks that integrates contour and color information into a compact vector. Loosely inspired by the human visual system and its mechanisms in efficiently identifying visual saliency, operations are performed on a fixed lattice of discrete positions by a set of edge detecting kernels that calculate region derivatives at different scales and orientation. The description method utilizes a weighted edge histogram where bins are populated on the premise of whether the regions contain edges belonging to the salient contours, while the discriminative power is further enhanced by integrating regional quantized color information. The proposed technique is both efficient and adaptive to the specifics of each depiction, while it does not need any training data to adjust parameters. Experimental evaluation conducted on seven benchmarking datasets against 13 well known global descriptors along with SIFT, SURF implementations (both in VLAD and BOVW), highlight the effectiveness and efficiency of the proposed descriptor.

Integration of global and local features based on hybrid similarity matching scheme for medical image retrieval system

Similarity measure is a challenging task in Content-Based Medical Image Retrieval (CBMIR) systems and the matching scheme is designed to improve the retrieval performance. However, there are several major shortcomings with conventional approaches for a matching scheme which can extensively affect their application of Medical Image Retrieval (MIR). To overcome the issues, in this paper a Multi-Level Matching (MLM) method for MIR using hybrid feature similarity is proposed. Here, images are represented by multi-level features including local level and global level. The Colour and Edge Directivity Descriptor (CEDD) is used as a colour and edge-based descriptor. Speeded-Up Robust Features (SURF) and Local Binary Pattern (LBP) are used as a local descriptor. The hybrid of both global and local features yields enhanced retrieval accuracy, which is analysed over collected image databases. From the experiment, the proposed method achieves better accuracy value about 92%, which is higher than other methods.

Content-based image retrieval model based on cost sensitive learning

How to retrieve the desired images quickly and accurately from the large scale image database has become a hot topic in the field of multimedia research. Many content-based image retrieval (CBIR) technologies already exist, but they are not always satisfactory. In many applications, the CBIR model based on machine learning relies heavily on the distance metric between samples. Although the traditional distance metric methods are simple and convenient, it is not always appropriate for CBIR tasks. In this paper, a novel distance metric learning (DML) method based on cost sensitive learning (CSL) is studied, and then it is used in a large margin distribution learning machine (LDM) to replace the traditional kernel functions. The improved LDM also takes into account CSL, and which is called CS-DLDM. Finally, CS-DLDM model is applied to CBIR tasks for implementation classification. We compare the proposed CS-DLDM model with other classifiers based on CSL. The experimental results show that the proposed CS-DLDM model not only has satisfactory classification performance but also the lowest misclassification cost, can effectively avoid the class imbalance of sample.

Content-based image retrieval with pachinko allocation model and a combination of colour, texture and text features

We present in this paper a content-based image retrieval system (CBIR) based on pachinko allocation model (PAM) and employing a combination of colour, texture and textual features. PAM is a probabilistic topic model which captures correlation not only between words in documents but also between different topics (concepts) responsible of their generation. Regardless this advantage of PAM, there is no work which explores its utility for content based image retrieval tasks. We aim to evaluate the use of PAM for CBIRs by implementing a system based on it. In this context, PAM was applied with two different modalities of features, image global features and textual indexes separately and combined. Mean average precision is evaluated. The use of PAM with combination of features has slightly improved results of using it with just one modality, this opens more perspective in order to enhance results.