Performance Of Image Retrieval Research Articles

An experimental evaluation of Siamese Neural Networks for robot localization using omnidirectional imaging in indoor environments

The objective of this paper is to address the localization problem using omnidirectional images captured by a catadioptric vision system mounted on the robot. For this purpose, we explore the potential of Siamese Neural Networks for modeling indoor environments using panoramic images as the unique source of information. Siamese Neural Networks are characterized by their ability to generate a similarity function between two input data, in this case, between two panoramic images. In this study, Siamese Neural Networks composed of two Convolutional Neural Networks (CNNs) are used. The output of each CNN is a descriptor which is used to characterize each image. The dissimilarity of the images is computed by measuring the distance between these descriptors. This fact makes Siamese Neural Networks particularly suitable to perform image retrieval tasks. First, we evaluate an initial task strongly related to localization that consists in detecting whether two images have been captured in the same or in different rooms. Next, we assess Siamese Neural Networks in the context of a global localization problem. The results outperform previous techniques for solving the localization task using the COLD-Freiburg dataset, in a variety of lighting conditions, specially when using images captured in cloudy and night conditions.

A Novel Semantic Content-Based Retrieval System for Hyperspectral Remote Sensing Imagery

With the growing use of hyperspectral remote sensing payloads, there has been a significant increase in the number of hyperspectral remote sensing image archives, leading to a massive amount of collected data. This highlights the need for an efficient content-based hyperspectral image retrieval (CBHIR) system to manage and enable better use of hyperspectral remote-sensing image archives. Conventional CBHIR systems characterize each image by a set of endmembers and then perform image retrieval based on pairwise distance measures. Such an approach significantly increases the computational complexity of the retrieval, mainly when the diversity of materials is high. Those systems also have difficulties in retrieving images containing particular materials with extremely low abundance compared to other materials, which leads to describing image content with inappropriate and/or insufficient spectral features. In this article, a novel CBHIR system to define global hyperspectral image representations based on a semantic approach to differentiate foreground and background image content for different retrieval scenarios is introduced to address these issues. The experiments conducted on a new benchmark archive of multi-label hyperspectral images, which is first introduced in this study, validate the retrieval accuracy and effectiveness of the proposed system. Comparative performance analysis with the state-of-the-art CBHIR systems demonstrates that modeling hyperspectral image content with foreground and background vocabularies has a positive effect on retrieval performance.

Composite Sketch+Text Queries for Retrieving Objects with Elusive Names and Complex Interactions

Non-native speakers with limited vocabulary often struggle to name specific objects despite being able to visualize them, e.g., people outside Australia searching for ‘numbats.’ Further, users may want to search for such elusive objects with difficult-to-sketch interactions, e.g., “numbat digging in the ground.” In such common but complex situations, users desire a search interface that accepts composite multimodal queries comprising hand-drawn sketches of “difficult-to-name but easy-to-draw” objects and text describing “difficult-to-sketch but easy-to-verbalize” object's attributes or interaction with the scene. This novel problem statement distinctly differs from the previously well-researched TBIR (text-based image retrieval) and SBIR (sketch-based image retrieval) problems. To study this under-explored task, we curate a dataset, CSTBIR (Composite Sketch+Text Based Image Retrieval), consisting of ~2M queries and 108K natural scene images. Further, as a solution to this problem, we propose a pretrained multimodal transformer-based baseline, STNet (Sketch+Text Network), that uses a hand-drawn sketch to localize relevant objects in the natural scene image, and encodes the text and image to perform image retrieval. In addition to contrastive learning, we propose multiple training objectives that improve the performance of our model. Extensive experiments show that our proposed method outperforms several state-of-the-art retrieval methods for text-only, sketch-only, and composite query modalities. We make the dataset and code available at: https://vl2g.github.io/projects/cstbir.

FFLDGA-Net: Image retrieval method based on Feature Fusion Learnable Descriptor Graph Attention Network

Image retrieval aims to retrieve and return the image in the database that is most similar to the query image. However, the performance of image retrieval models is often hindered by the limited dimensionality of images, which lacks depth information about objects. To address this issue, we propose a novel image retrieval model called FFLDGA-Net (Feature Fusion-based Learnable Descriptor Graph Attention Network). This model aims to overcome the absence of depth information in images by fusing feature information from both image data and point cloud data. First, we introduce the LDGA-Net, which effectively improved the model’s ability to mine hard samples and negative samples. Then, we combine the multi-scale route convolution module with a one-dimensional path aggregation network to fuse point clouds and image features at multiple scales, and establish a high-dimensional relationship between features and low-dimensional features. To mitigate training noise, we incorporate a soft label strategy tailored to the dataset’s characteristics. Our experimental results on two benchmark datasets demonstrate the significant improvements achieved by FFLDGA-Net in image retrieval performance.

Evaluation of retrieval accuracy and visual similarity in content-based image retrieval of chest CT for obstructive lung disease

The aim of our study was to assess the performance of content-based image retrieval (CBIR) for similar chest computed tomography (CT) in obstructive lung disease. This retrospective study included patients with obstructive lung disease who underwent volumetric chest CT scans. The CBIR database included 600 chest CT scans from 541 patients. To assess the system performance, follow-up chest CT scans of 50 patients were evaluated as query cases, which showed the stability of the CT findings between baseline and follow-up chest CT, as confirmed by thoracic radiologists. The CBIR system retrieved the top five similar CT scans for each query case from the database by quantifying and comparing emphysema extent and size, airway wall thickness, and peripheral pulmonary vasculatures in descending order from the database. The rates of retrieval of the same pairs of query CT scans in the top 1–5 retrievals were assessed. Two expert chest radiologists evaluated the visual similarities between the query and retrieved CT scans using a five-point scale grading system. The rates of retrieving the same pairs of query CTs were 60.0% (30/50) and 68.0% (34/50) for top-three and top-five retrievals. Radiologists rated 64.8% (95% confidence interval 58.8–70.4) of the retrieved CT scans with a visual similarity score of four or five and at least one case scored five points in 74% (74/100) of all query cases. The proposed CBIR system for obstructive lung disease integrating quantitative CT measures demonstrated potential for retrieving chest CT scans with similar imaging phenotypes. Further refinement and validation in this field would be valuable.

Locating Target Regions for Image Retrieval in an Unsupervised Manner.

Image retrieval performance can be improved by training a convolutional neural network (CNN) model with annotated data to facilitate accurate localization of target regions. However, obtaining sufficiently annotated data is expensive and impractical in real settings. It is challenging to achieve accurate localization of target regions in an unsupervised manner. To address this problem, we propose a new unsupervised image retrieval method named unsupervised target region localization (UTRL) descriptors. It can precisely locate target regions without supervisory information or learning. Our method contains three highlights: 1) we propose a novel zero-label transfer learning method to address the problem of co-localization in target regions. This enhances the potential localization ability of pretrained CNN models through a zero-label data-driven approach; 2) we propose a multiscale attention accumulation method to accurately extract distinguishable target features. It distinguishes the importance of features by using local Gaussian weights; and 3) we propose a simple yet effective method to reduce vector dimensionality, named twice-PCA-whitening (TPW), which reduces the performance degradation caused by feature compression. Notably, TPW is a robust and general method that can be widely applied to image retrieval tasks to improve retrieval performance. This work also facilitates the development of image retrieval based on short vector features. Extensive experiments on six popular benchmark datasets demonstrate that our method achieves about 7% greater mean average precision (mAP) compared to existing state-of-the-art unsupervised methods.

Deep Progressive Asymmetric Quantization Based on Causal Intervention for Fine-Grained Image Retrieval

In the field of computer vision, fine-grained image retrieval is an extremely challenging task due to the inherently subtle intra-class object variations. In addition, the high-dimensional real-valued features extracted from large-scale fine-grained image datasets slow the retrieval speed and increase the storage cost. To solve above issues, existing fine-grained image retrieval methods mainly focus on finding more discriminative local regions for generating discriminative and compact hash codes, which achieve limited fine-grained image retrieval performance due to the large quantization errors and the confounding granularities and context of discriminative parts, i.e., the correct recognition of fine-grained objects mainly attribute to the discriminative parts and their context. To learn robust causal features and reduce the quantization errors, we propose a deep progressive asymmetric quantization (DPAQ) method based on causal intervention to learn compact and robust descriptions for fine-grained image retrieval task. Specifically, we introduce a structural causal model to learn robust casual features via causal intervention for fine-grained visual recognition. Subsequently, we design a progressive asymmetric quantization layer in the feature embedding space, which can preserve the semantic information and reduce the quantization errors sufficiently. Finally, we incorporate both the fine-grained image classification and retrieval tasks into an end-to-end deep learning architecture for generating robust and compact descriptions. Experimental results on several fine-grained image retrieval datasets demonstrate that the proposed DPAQ method performs the best for fine-grained image retrieval task and surpasses the state-of-the art fine-grained hashing methods by a large margin.

Image retrieval using unsupervised prompt learning and regional attention

Identifying the target object in an image can produce more accurate and discriminative feature representations, which cansignificantlyimprove large-scale instance-level image-retrieval performance. However, it is usually difficult to obtain annotation information for all target objects in a dataset manually, which makes it challenging to automatically identify target objects. To address this issue, we propose a novel method of instance-level image retrieval based on unsupervised prompt learning andregionalattention (PLRA) rather than manual annotation. It includes three main components: (1) We propose an unsupervised prompt learning method to identify an image’s target object. It reconstructs deep features by mining prompt information, then designs prompt factors to identify the target object based on the reconstructed features. (2) We propose a new regional attention method to extract the distinguishing features of the target object. This method captures important feature regions in four dimensions: global, local, spatial, and channel, which improves the diversity and discriminability of the representation. (3) We propose a general hybrid PCA-whitening (HPW) method based on multi-parameter learning and feature fusion, which trades off feature dimensionality with retrieval performance. This method significantly improves performance and reduces vector dimensionality in a plug-and-play manner. Comprehensive experiments on five benchmark datasets show that the proposed method significantly outperforms existing state-of-the-art methods based on unsupervised feature aggregation.

Content-based image retrieval for medical diagnosis using fuzzy clustering and deep learning

Brain tumors are one of the most threatening causes of death worldwide. Neuroradiologists widely use Magnetic Resonance Imaging as a golden standard non-invasive imaging method for diagnosing Glioma grades. Brain tumor diagnosis based on radiology images is tedious and highly subjective to intra and inter-observer variability; it adversely affects therapeutic planning. Better medical treatment can be provided relating to similar past cases if relevant images are retrieved from an extensive medical image database. Content-Based Image Retrieval systems are potent image tools for dealing with such massive datasets. In a CBIR system, accurate classification and retrieval of similar pathological images can be effectively automated by utilising Convolutional Neural Network-based feature extraction methods. This paper presents a Content-Based Medical Image Retrieval pipeline on a medical domain using the CNN model for feature extraction and the clustering method used to index the feature map database. The proposed system applies a Multi-level Gain-based feature selection to reduce the dimensionality of the feature vectors obtained from the pre-trained CNN models. The experiment follows five-fold cross-validation on BraTS 2018 and 2020 datasets. The proposed system achieves state-of-the-art MRI brain image retrieval performance with a mean Average Precision and Precision@10 of 98.15% and 97.62%, respectively. The results prove the proposed method’s effectiveness and validate its clinical application.

Image Retrieval based on Deep Learning

With the rapid growth of image data, how to efficiently and accurately extract useful features from massive image data and perform fast image retrieval has become an important research direction. This study focuses on the design and training of deep learning-based image feature extraction networks to improve the robustness and generalization of image features by optimizing the network structure and loss function. In order to evaluate the performance of the system, this study also designs appropriate evaluation indicators and conducts corresponding experiments. Through experimental verification, the results show that these methods can effectively improve the performance of image feature extraction and image retrieval, and have broad potential in practical applications.

Image Retrieval Performance Tuning Using Optimization Algorithms

NThe relentlessness of modern life's pace has pushed many people to look for ways to save time so they may continue living the way they choose. The management of traffic presents a considerable obstacle due to the fact that a large proportion of persons have difficulties associated with transportation. As a result, fixing traffic problems becomes an absolute need. Machine learning stands out as a useful resource in this setting, providing deeper understanding and better analytical tools for sifting through complicated statistical data. The feasibility of travel for both large and light vehicles may be quickly assessed by experts, allowing for more timely and well-informed decisions. These evaluations serve as the basis for developing separate roadways, laws and sets of rules for various classes of vehicles. This study examines six characteristics of vehicular traffic and travel situations over 38,114 occurrences. Our goal is to improve traffic management through prediction and optimization using six different optimizers from the deep learning area. Based on observed patterns of truck traffic over a certain time period, these methods help determine optimal distribution paths for sent commodities. Six different deep learning optimizer models are compared and contrasted in this study. The objective is to use these examples to determine which optimizer is best. It's not easy to pick the best optimizer for use in deep learning. To this goal, we conducted an in-depth analysis of six industry-leading optimizers to identify the best tool for predicting traffic accidents. We ran extensive tests using a dataset that had 30,492 training examples (80%) and 7,622 testing instances (20%). Different seed values, ranging from 20 to 100, were used in each iteration of the experiment. We tested and compared the following optimizers: the Adaptive Gradient (AG) Algorithm, the Adaptive Learning Rate (ALR) Method, the Root Mean Squared (RMS) Propagation, the Adaptive Moment (AM) Estimation, the Nesterov-accelerated Adaptive Moment (NAM) Estimation, and the Stochastic Gradient (SG) Descent, taking into account processing times, prediction accuracy, and error analysis. The results of the experiment showed that the NAM Estimation Optimizer was much superior. Time spent processing data was cut down, and errors were kept to a minimum (0.03%). Prediction accuracy was also exceptionally high at 99.85%. This result reaffirms NAM Estimation's promise as a leading method for improving traffic management and making accurate trip predictions.

Discriminative Geometric-Structure-Based Deep Hashing for Large-Scale Image Retrieval.

Deep hashing reaps the benefits of deep learning and hashing technology, and has become the mainstream of large-scale image retrieval. It generally encodes image into hash code with feature similarity preserving, that is, geometric-structure preservation, and achieves promising retrieval results. In this article, we find that existing geometric-structure preservation manner inadequately ensures feature discrimination, while improving feature discrimination of hash code essentially determines hash learning retrieval performance. This fact principally spurs us to propose a discriminative geometric-structure-based deep hashing method (DGDH), which investigates three novel loss terms based on class centers to induce the so-called discriminative geometrical structure. In detail, the margin-aware center loss assembles samples in the same class to the corresponding class centers for intraclass compactness, then a linear classifier based on class center serves to boost interclass separability, and the radius loss further puts different class centers on a hypersphere to tentatively reduce quantization errors. An efficient alternate optimization algorithm with guaranteed desirable convergence is proposed to optimize DGDH. We theoretically analyze the robustness and generalization of the proposed method. The experiments on five popular benchmark datasets demonstrate superior image retrieval performance of the proposed DGDH over several state of the arts.

Aurora retrieval in all-sky images based on hash vision transformer

Auroras are bright occurrences when high-energy particles from the magnetosphere and solar wind enter Earth's atmosphere through the magnetic field and collide with atoms in the upper atmosphere. The morphological and temporal characteristics of auroras are essential for studying large-scale magnetospheric processes. While auroras are visible to the naked eye from the ground, scientists use deep learning algorithms to analyze all-sky images to understand this phenomenon better. However, the current algorithms face challenges due to inefficient utilization of global features and neglect the excellent fusion of local and global feature representations extracted from aurora images. Hence, this paper introduces a Hash-Transformer model based on Vision Transformer for aurora retrieval from all-sky images. Experimental results based on real-world data demonstrate that the proposed method effectively improves aurora image retrieval performance. It provides a new avenue to study aurora phenomena and facilitates the development of related fields.

Towards Food Image Retrieval via Generalization-Oriented Sampling and Loss Function Design

Food computing has increasingly received widespread attention in the multimedia field. As a basic task of food computing, food image retrieval has wide applications, that is, food image retrieval can help users to find the desired food from a large number of food images. Besides, the retrieved information can be applied to establish a richer database for the subsequent food content-related recommendation. Food image retrieval aims to achieve better performance on novel categories. Thus, it is worth studying to transfer the embedding ability from the training set to the unseen test set, that is, the generalization of the model. Food is influenced by various factors, such as culture and geography, leading to great differences between domains, such as Asian food and western food. Therefore, it is challenging to study the generalization of the model in food image retrieval. In this article, we improve the classical metric learning framework and propose a generalization-oriented sampling strategy, which boosts the generalization of the model by maximizing the intra-class distance from a proportion of positive pairs to avoid the excessive distance compression in the embedding space. Considering that the existing optimization process is in an opposite direction to our proposed sampling strategy, we further propose an adaptive gradient assignment policy named gradient-adaptive optimization , which can alleviate the intra-class distance compression during optimization by assigning different gradients to different samples. Extensive evaluation on three popular food image datasets demonstrates the effectiveness of the proposed method. We also experiment on three popular general datasets to prove that solving the problem from the generalization can also improve the performance of general image retrieval. Code is available at https://github.com/Jiajun-ISIA/Generalization-oriented-Sampling-and-Loss .

An evaluation metric for image retrieval systems, using entropy for grouped precision of relevant retrievals

Evaluating the performance of Content-Based Image Retrieval (CBIR) systems is a challenging and intricate task, even for experts in the field. The literature presents a vast array of CBIR systems, each applied to various image databases. Traditionally, automatic metrics employed for CBIR evaluation have been borrowed from the Text Retrieval (TR) domain, primarily precision and recall metrics. However, this paper introduces a novel quantitative metric specifically designed to address the unique characteristics of CBIR. The proposed metric revolves around the concept of grouping relevant images and utilizes the entropy of the retrieved relevant images. Grouping together relevant images holds great value from a user perspective, as it enables more coherent and meaningful results. Consequently, the metric effectively captures and incorporates the grouping of the most relevant outcomes, making it highly advantageous for CBIR evaluation. Additionally, the proposed CBIR metric excels in differentiating between results that might appear similar when assessed using other metrics. It exhibits a superior ability to discern subtle distinctions among retrieval outcomes. This enhanced discriminatory power is a significant advantage of the proposed metric. Furthermore, the proposed performance metric is designed to be straightforward to comprehend and implement. Its simplicity and ease of use contribute to its practicality for researchers and practitioners in the field of CBIR. To validate the effectiveness of our metric, we conducted a comprehensive comparative study involving prominent and well-established CBIR evaluation metrics. The results of this study demonstrate that our proposed metric exhibits robust discrimination power, outperforming existing metrics in accurately evaluating CBIR system performance.

Developing a model semantic‐based image retrieval by combining KD‐Tree structure with ontology

AbstractThe paper proposes an alternative approach to improve the performance of image retrieval. In this work, a framework for image retrieval based on machine learning and semantic retrieval is proposed. In the preprocessing phase, the image is segmented objects by using Graph‐cut, and the feature vectors of objects presented in the image and their visual relationships are extracted using R‐CNN. The feature vectors, visual relationships, and their symbolic labels are stored in KD‐Tree data structures which can be used to predict the label of objects and visual relationships later. To facilitate semantic query, the images use the RDF data model and create an ontology for the symbolic labels annotated. For each query image, after extracting their feature vectors, the KD‐Tree is used to classify the objects and predict their relationship. After that, a SPARQL query is built to extract a set of similar images. The SPARQL query consists of triple statements describing the objects and their relationship which were previously predicted. The evaluation of the framework with the MS‐COCO dataset and Flickr showed that the precision achieved scores of 0.9218 and 0.9370, respectively.

Unsupervised Hashing with Semantic Concept Mining

Recently, to improve the unsupervised image retrieval performance, plenty of unsupervised hashing methods have been proposed by designing a semantic similarity matrix, which is based on the similarities between image features extracted by a pre-trained CNN model. However, most of these methods tend to ignore high-level abstract semantic concepts contained in images. Intuitively, concepts play an important role in calculating the similarity among images. In real-world scenarios, each image is associated with some concepts, and the similarity between two images will be larger if they share more identical concepts. Inspired by the above intuition, in this work, we propose a novel Unsupervised Hashing with Semantic Concept Mining, called UHSCM, which leverages a VLP model to construct a high-quality similarity matrix. Specifically, a set of randomly chosen concepts is first collected. Then, by employing a vision-language pretraining (VLP) model with the prompt engineering which has shown strong power in visual representation learning, the set of concepts is denoised according to the training images. Next, the proposed method UHSCM applies the VLP model with prompting again to mine the concept distribution of each image and construct a high-quality semantic similarity matrix based on the mined concept distributions. Finally, with the semantic similarity matrix as guiding information, a novel hashing loss with a modified contrastive loss based regularization item is proposed to optimize the hashing network. Extensive experiments on three benchmark datasets show that the proposed method outperforms the state-of-the-art baselines in the image retrieval task.

Learning image representations for content-based image retrieval of radiotherapy treatment plans

Objective. In this work, we propose a content-based image retrieval (CBIR) method for retrieving dose distributions of previously planned patients based on anatomical similarity. Retrieved dose distributions from this method can be incorporated into automated treatment planning workflows in order to streamline the iterative planning process. As CBIR has not yet been applied to treatment planning, our work seeks to understand which current machine learning models are most viable in this context. Approach. Our proposed CBIR method trains a representation model that produces latent space embeddings of a patient’s anatomical information. The latent space embeddings of new patients are then compared against those of previous patients in a database for image retrieval of dose distributions. All source code for this project is available on github. Main results. The retrieval performance of various CBIR methods is evaluated on a dataset consisting of both publicly available image sets and clinical image sets from our institution. This study compares various encoding methods, ranging from simple autoencoders to more recent Siamese networks like SimSiam, and the best performance was observed for the multitask Siamese network. Significance. Our current results demonstrate that excellent image retrieval performance can be obtained through slight changes to previously developed Siamese networks. We hope to integrate CBIR into automated planning workflow in future works.

An optimized deep supervised hashing model for fast image retrieval

As multimedia data grows exponentially, searching for and retrieving a relevant image is becoming a challenge for researchers. Hashing is a widely adopted method because of its high performance in image retrieval with deep neural networks and multiple convolutional layers. Even so, most hashing methods ignore the computational cost and memory storage consumption. When the deep hashing model size is large, it leads to a slowdown in the response time of the model compared to the small model. Addressing these issues, a novel optimized deep supervised hashing based on a teacher-student approach for swift and precise image retrieval is proposed in this paper. In this work, the small student model is trained using the knowledge distillation from the large teacher model and the information from the one-hot labels. Therefore, a weight allocation loss function based on the teacher and student models is defined. Meanwhile, we apply model pruning to decrease the amount of the student model further to increase the response time. Therefore, knowledge distillation is performed on the pruned model. After that, the remaining weights are quantized to reach the smaller size of the model. Extensive experimental outcomes on two widely used datasets prove the outstanding efficiency of our proposed method.

Design of Extensible Structured Interferometric Array Utilizing the “Coarray” Concept

The optimum placement of receiving telescope antennas is a central topic for designing radio interferometric arrays, and this determines the performance of the obtained information. A variety of arrays are designed for different purposes, and they perform poorly in scalability. In this paper, we consider a subclass of structured sparse arrays, namely nested arrays, and examine the important role of “coarray” in interferometric synthesis imaging, which is utilized to design nested array configurations for a complete uniform Fourier plane coverage in both supersynthesis and instantaneous modes. Both nested arrays and the theory of the coarray have rich research achievements, and we apply them to astronomy to design arrays with good scalability and imaging performance. Simulated celestial source image retrieval performance validates the effectiveness of nested interferometric arrays.