Image Retrieval Method Research Articles

TPTE: Text-Guided Patch Token Exploitation for Unsupervised Fine-Grained Representation Learning

Recent advances in pre-trained vision-language models have successfully boosted the performance of unsupervised image representation in many vision tasks. Most of existing works focus on learning global visual features with Transformers and neglect detailed local cues, leading to suboptimal performance in fine-grained vision tasks. In this article, we propose a text-guided patch token exploitation framework to enhance the discriminative power of unsupervised representation by exploiting more detailed local features. Our text-guided decoder extracts local features with the guidance of texts or learned prompts describing discriminative object parts. We hence introduce a local-global relation distillation loss to promote the joint optimization of local and global features. The proposed method allows to flexibly extract either global or global-local features as the image representation. It significantly outperforms previous methods in fine-grained image retrieval and base-to-new fine-grained classification tasks. For instance, our Recall@1 metric surpasses the recent unsupervised retrieval method STML by 6.0% on the SOP dataset. The code is publicly available at https://github.com/maosnhehe/TPTE .

Deep hashing and attention mechanism-based image retrieval of osteosarcoma scans for diagnosis of bone cancer

BackgroundDue to its intricate nature and substantial data size, microscopic image data of osteosarcoma often present a significant obstacle to the effectiveness of conventional image retrieval methods. Therefore, this study investigates a new approach for medical image retrieval using advanced deep hashing techniques and attention mechanisms to address these challenges more effectively. MethodThe proposed algorithm significantly improves osteosarcoma cell microscopic image retrieval efficiency and accuracy using deep hashing and attention mechanisms. Image preprocessing includes adaptive histogram equalization and dataset augmentation to enhance quality and diversity. Feature extraction employs the WRN-AM model to map high-dimensional features to a low-dimensional hash code space, improving retrieval efficiency. Finally, similarity matching via Hamming distance allows rapid and precise identification of similar images. ResultsThe study shows notable advancements: the WRN-AM model achieves 93.2% classification accuracy and 97.09% mAP using 64-bit hash codes. These findings underscore the technique’s effective performance in extracting and categorizing diverse microscopic cell data efficiently and reliably. ConclusionsThis innovative approach provides a robust solution for retrieving and classifying microscopic data of osteosarcoma cells and other cell types, speeding up clinical diagnosis and medical research. It facilitates quicker access and analysis of patient image data, enhancing diagnostic precision and treatment planning for healthcare professionals. Concurrently, it supports researchers in leveraging medical image data more efficiently, fostering progress and innovation in the medical field.

SMedIR: secure medical image retrieval framework with ConvNeXt-based indexing and searchable encryption in the cloud

The security and privacy of medical images are crucial due to their sensitive nature and the potential for severe consequences from unauthorized modifications, including data breaches and inaccurate diagnoses. This paper introduces a method for lossless medical image retrieval from encrypted images stored on third-party clouds. The proposed approach employs a symmetric integrity-centric image encryption scheme, leveraging multiple chaotic maps and cryptographic hash techniques, to ensure lossless image reconstruction. Medical images are first encrypted by the image owners and converted into hashcodes encapsulating essential features using a deep hashing technique with the ConvNeXt network as the backbone in parallel. To ensure index privacy, these hashcodes are encrypted in a searchable manner. The encrypted medical images, along with a secure index, are subsequently uploaded to cloud storage. Authorized medical image users can request similar medical images for diagnostic purposes by submitting a query image, from which a search trapdoor is generated and sent to the cloud. The retrieval process involves a secure similar image search over the encrypted indexes, followed by decryption along with integrity verification of the retrieved images. The proposed method has been rigorously tested on three standard medical datasets, demonstrating an improvement of 5-20% in retrieval accuracy compared to standard baselines. Formal security analysis and experimental results indicate that the proposed scheme offers enhanced security and retrieval accuracy, making it an effective solution for the encrypted storage and secure retrieval of medical image data.

Backward induction-based deep image search.

Conditional image retrieval (CIR), which involves retrieving images by a query image along with user-specified conditions, is essential in computer vision research for efficient image search and automated image analysis. The existing approaches, such as composed image retrieval (CoIR) methods, have been actively studied. However, these methods face challenges as they require either a triplet dataset or richly annotated image-text pairs, which are expensive to obtain. In this work, we demonstrate that CIR at the image-level concept can be achieved using an inverse mapping approach that explores the model's inductive knowledge. Our proposed CIR method, called Backward Search, updates the query embedding to conform to the condition. Specifically, the embedding of the query image is updated by predicting the probability of the label and minimizing the difference from the condition label. This enables CIR with image-level concepts while preserving the context of the query. In this paper, we introduce the Backward Search method that enables single and multi-conditional image retrieval. Moreover, we efficiently reduce the computation time by distilling the knowledge. We conduct experiments using the WikiArt, aPY, and CUB benchmark datasets. The proposed method achieves an average mAP@10 of 0.541 on the datasets, demonstrating a marked improvement compared to the CoIR methods in our comparative experiments. Furthermore, by employing knowledge distillation with the Backward Search model as the teacher, the student model achieves a significant reduction in computation time, up to 160 times faster with only a slight decrease in performance. The implementation of our method is available at the following URL: https://github.com/dhlee-work/BackwardSearch.

Cross-modality sub-image retrieval using contrastive multimodal image representations

In tissue characterization and cancer diagnostics, multimodal imaging has emerged as a powerful technique. Thanks to computational advances, large datasets can be exploited to discover patterns in pathologies and improve diagnosis. However, this requires efficient and scalable image retrieval methods. Cross-modality image retrieval is particularly challenging, since images of similar (or even the same) content captured by different modalities might share few common structures. We propose a new application-independent content-based image retrieval (CBIR) system for reverse (sub-)image search across modalities, which combines deep learning to generate representations (embedding the different modalities in a common space) with robust feature extraction and bag-of-words models for efficient and reliable retrieval. We illustrate its advantages through a replacement study, exploring a number of feature extractors and learned representations, as well as through comparison to recent (cross-modality) CBIR methods. For the task of (sub-)image retrieval on a (publicly available) dataset of brightfield and second harmonic generation microscopy images, the results show that our approach is superior to all tested alternatives. We discuss the shortcomings of the compared methods and observe the importance of equivariance and invariance properties of the learned representations and feature extractors in the CBIR pipeline. Code is available at: https://github.com/MIDA-group/CrossModal_ImgRetrieval.

Plaid Fabric Image Retrieval Based on Wavelet Transform and SIFT Features

Abstract This study presents a method for plaid fabric image retrieval that combines wavelet and SIFT features to address the challenges of accuracy and efficiency in fabric retrieval due to diverse fabric types. The process starts with cropping plaid fabric images and applying histogram equalization to improve brightness and contrast. Texture is enhanced using the Sobel operator, and the Haar wavelet transform extracts image high-frequency components in various directions. Wavelet features are then derived through histogram statistics. The SIFT algorithm is utilized to describe local features by capturing key points and directional information. A codebook aggregates these features from the fabric database, and VLAD encoding generates a vector for the image features, which is further reduced to 256 dimensions via PCA. A similarity-weighted fusion method combines the wavelet and SIFT features, achieving an mAP of 0.67 and an average retrieval time of 1.1 seconds per image. This method significantly enhances plaid fabric retrieval, aiding in fabric design and production.

Central similarity consistency hashing for asymmetric image retrieval

Asymmetric image retrieval methods have drawn much attention due to their effectiveness in resource-constrained scenarios. They try to learn two models in an asymmetric paradigm, i.e., a small model for the query side and a large model for the gallery. However, we empirically find that the mutual training scheme (learning with each other) will inevitably degrade the performance of the large gallery model, due to the negative effects exerted by the small query one. In this paper, we propose Central Similarity Consistency Hashing (CSCH), which simultaneously learns a small query model and a large gallery model in a mutually promoted manner, ensuring both high retrieval accuracy and efficiency on the query side. To achieve this, we first introduce heuristically generated hash centers as the common learning target for both two models. Instead of randomly assigning each hash center to its corresponding category, we introduce the Hungarian algorithm to optimally match each of them by aligning the Hamming similarity of hash centers to the semantic similarity of their classes. Furthermore, we introduce the instance-level consistency loss, which enables the explicit knowledge transfer from the gallery model to the query one, without the sacrifice of gallery performance. Guided by the unified learning of hash centers and the distilled knowledge from gallery model, the query model can be gradually aligned to the Hamming space of the gallery model in a decoupled manner. Extensive experiments demonstrate the superiority of our CSCH method compared with current state-of-the-art deep hashing methods. The open-source code is available at https://github.com/dubanx/CSCH.

An Image-Retrieval Method Based on Cross-Hardware Platform Features

Artificial intelligence (AI) models have already achieved great success in fields such as computer vision and natural language processing. However, deploying AI models based on heterogeneous hardware is difficult to ensure accuracy consistency, especially for precision sensitive feature-based image retrieval. In this article, we realize an image-retrieval method based on cross-hardware platform features, aiming to prove that the features of heterogeneous hardware platforms can be mixed, in which the Huawei Atlas 300V and NVIDIA TeslaT4 are used for experiments. First, we compared the decoding differences of heterogeneous hardware, and used CPU software decoding to help hardware decoding improve the decoding success rate. Then, we compared the difference between the Atlas 300V and TeslaT4 chip architectures and tested the differences between the two platform features by calculating feature similarity. In addition, the scaling mode in the pre-processing process was also compared to further analyze the factors affecting feature consistency. Next, the consistency of capture and correlation based on video structure were verified. Finally, the experimental results reveal that the feature results from the TeslaT4 and Atlas 300V can be mixed for image retrieval based on cross-hardware platform features. Consequently, cross-platform image retrieval with low error is realized. Specifically, compared with the Atlas 300V hard and CPU soft decoding, the TeslaT4 hard decoded more than 99% of the image with a decoding pixel maximum difference of +1/−1. From the average of feature similarity, the feature similarity between the Atlas 300V and TeslaT4 exceeds 99%. The difference between the TeslaT4 and Atlas 300V in recall and mAP in feature retrieval is less than 0.1%.

Iris image retrieval using partial matching of image blocks

AbstractThe problem of human identification through recognition of patterns in iris images captured in unconstrained environments results in image artefacts such as image occlusion and specular reflection, in which iris tissue is observable to extremely low content. To overcome this problem, this paper presents a novel method for iris image retrieval and recognition based on partial pattern matching. The main contribution of the proposed method relies on an image partitioning schema in which the iris is divided into non‐overlapping blocks with varying dimensions, facilitating the identification and removal of image regions impaired by eyelids, eyelashes, and specular reflections. In fact, the blocks that contain artefacts are completely ignored and those blocks are preserved that include useful patterns for identification. In addition, a multi‐feature similarity followed by a score fusion technique is employed for ranking the retrieval results. The remarkable results of the classification stage include an accuracy of 100%, 98.75%, and 99.94% on three benchmark databases, including UPOL, UBIRIS.V2, and CASIA‐Iris‐Interval.v3, respectively. Additionally, in the retrieval stage, the proposed method achieves a precision of 100% on all three benchmark databases, a recall of 83.33%, and a F1‐measure of 90.91 on the CASIA‐Iris‐Interval.v3 dataset.

Multi-FusNet: fusion mapping of features for fine-grained image retrieval networks.

As the diversity and volume of images continue to grow, the demand for efficient fine-grained image retrieval has surged across numerous fields. However, the current deep learning-based approaches to fine-grained image retrieval often concentrate solely on the top-layer features, neglecting the relevant information carried in the middle layer, even though these information contains more fine-grained identification content. Moreover, these methods typically employ a uniform weighting strategy during hash code mapping, risking the loss of critical region mapping-an irreversible detriment to fine-grained retrieval tasks. To address the above problems, we propose a novel method for fine-grained image retrieval that leverage feature fusion and hash mapping techniques. Our approach harnesses a multi-level feature cascade, emphasizing not just top-layer but also intermediate-layer image features, and integrates a feature fusion module at each level to enhance the extraction of discriminative information. In addition, we introduce an agent self-attention architecture, marking its first application in this context, which steers the model to prioritize on long-range features, further avoiding the loss of critical regions of the mapping. Finally, our proposed model significantly outperforms existing state-of-the-art, improving the retrieval accuracy by an average of 40% for the 12-bit dataset, 22% for the 24-bit dataset, 16% for the 32-bit dataset, and 11% for the 48-bit dataset across five publicly available fine-grained datasets. We also validate the generalization ability and performance stability of our proposed method by another five datasets and statistical significance tests. Our code can be downloaded from https://github.com/BJFU-CS2012/MuiltNet.git.

Performance evaluation of attention-deep hashing based medical image retrieval in brain MRI datasets

BackgroundBrain MRI images pose significant challenges due to their complexity and voluminous data, which often hinder the accuracy of traditional image retrieval methods. In response, this research delves into a novel medical image retrieval approach grounded in attention mechanisms and deep hashing techniques. MethodologyThe study enhances the network's edge perception capability by incorporating a dual mixed attention module (CBAM-MA dual attention) into the convolutional neural network architecture. This addition optimally captures edge saliency and distributional features by introducing an extra attention layer atop CBAM. Furthermore, a novel triple loss function is introduced, amalgamating Euclidean and cosine distances. This fusion exploits the strengths of both distance metrics to comprehensively consider sample geometric attributes, thereby generating more robust and expressive feature representations crucial for preserving categorical details. ResultsExperimental evaluations showcase significant performance advancements in MRI image retrieval tasks using the proposed method. The average hit rate stands at 0.7783, average precision at 0.752246, and average inverse rank at 0.956721. These metrics collectively underscore the method's efficacy in enhancing the quality and expediting the diagnosis of brain diseases. ConclusionThe research underscores the critical role of attention and deep hashing techniques in augmenting the accuracy and efficiency of medical image retrieval, particularly in the context of brain MRI images. The proposed method's notable performance improvements pave the way for enhanced diagnostic capabilities, signaling a significant step forward in improving healthcare outcomes related to brain diseases.

A Transformer and Visual Foundation Model-Based Method for Cross-View Remote Sensing Image Retrieval

Abstract. Retrieving UAV images that lack POS information with georeferenced satellite orthoimagery is challenging due to the differences in angles of views. Most existing methods rely on deep neural networks with a large number of parameters, leading to substantial time and financial investments in network training. Consequently, these methods may not be well-suited for downstream tasks that have high timeliness requirements. In this work, we propose a cross-view remote sensing image retrieval method based on transformer and visual foundation model. We investigated the potential of visual foundation model for extracting common features from cross-view images. Training is only conducted on a small, self-designed retrieval head, alleviating the burden of network training. Specifically, we designed a CVV module to optimize the features extracted from the visual foundation model, making these features more adept for cross-view image retrieval tasks. And we designed an MLP head to achieve similarity discrimination. The method is verified on a publicly available dataset containing multiple scenes. Our method shows excellent results in terms of both efficiency and accuracy on 15 sub-datasets (10 or 50 scene categories) derived from the public dataset, which holds practical value in engineering applications with streamlined scene categories and constrained computational resources. Furthermore, we initiated a comprehensive discussion and conducted ablation experiments on the network design to validate its efficacy. Additionally, we analyzed the presence of overfitting within the network and deliberated on the limitations of our study, proposing potential avenues for future enhancements.

A novel deep learning based CBIR model using Convolutional Siamese Neural Networks

Content-Based Image Retrieval (CBIR) is a technique that involves retrieving similar images from a large database by analysing the content features of the query image. The heavy usage of digital platforms and devices has in a way promoted CBIR and its allied technologies in computer vision and artificial intelligence. The process entails comparing the representative features of the query image with those of the images in the dataset to rank them for retrieval. Past research was centered around handcrafted feature descriptors based on traditional visual features. But with the advent of deep learning the traditional manual method of feature engineering gave way to automatic feature extraction. In this study, a cascaded network is utilised for CBIR. In the first stage, the model employs multi-modal features from variational autoencoders and super-pixelated image characteristics to narrow down the search space. In the subsequent stage, an end-to-end deep learning network known as a Convolutional Siamese Neural Network (CSNN) is used. The concept of pseudo-labeling is incorporated to categorise images according to their affinity and similarity with the query image. Using this pseudo-supervised learning approach, this network evaluates the similarity between a query image and available image samples. The Siamese network assigns a similarity score to each target image, and those that surpass a predefined threshold are ranked and retrieved. The suggested CBIR system undergoes testing on a widely recognized public dataset: the Oxford dataset and its performance is measured against cutting-edge image retrieval methods. The findings reveal substantial enhancements in retrieval performance in terms of several standard benchmarks such as average precision, average error rate, average false positive rate etc., providing strong support for utilising images from interconnected devices.

ANALYZING RETRIEVAL METHOD USING CLASSIFICATION AND CLUSTERING PROCESS OF REMOTE SENSOR SATELLITE IMAGES

Today, remote sensors and satellite images are utilized across enormous regions. As visual tools, they are very important in cities, agriculture, disaster relief, and tracking systems. Pattern recognition, classification, and clustering are examples of machine learning techniques that can be used to acquire and retrieve the images. This is because they can accurately categorize and classify the images into clusters, which will help search engines discover the image that was given as a question. In our study, we look at Naïve Bayes, SVM Linear and Non-Linear, and Random Forest as classification methods, and Agglomerative Hierarchy and BIRCH as clustering methods. In this study, the fast recovery method and high accuracy rate of image retrieval by classifiers from machine learning were compared with clustering techniques from transfer learning. Using the UC Merced dataset, this research demonstrates that the accuracy classifier as SVM linear is 64.6%, SVM non-linear is 77.7%, random forest is 66.8%, and naïve bayes is 60.3%, and clustering of Agglomerative is 96.5% and BIRCH is 92.8%. The work that was mentioned is one of a kind because the clustering methods work better than the classification methods when they are used. It's unsupervised learning whose scores are above 90% for all classes.

Efficient Image Retrieval Using Hierarchical K-Means Clustering.

The objective of content-based image retrieval (CBIR) is to locate samples from a database that are akin to a query, relying on the content embedded within the images. A contemporary strategy involves calculating the similarity between compact vectors by encoding both the query and the database images as global descriptors. In this work, we propose an image retrieval method by using hierarchical K-means clustering to efficiently organize the image descriptors within the database, which aims to optimize the subsequent retrieval process. Then, we compute the similarity between the descriptor set within the leaf nodes and the query descriptor to rank them accordingly. Three tree search algorithms are presented to enable a trade-off between search accuracy and speed that allows for substantial gains at the expense of a slightly reduced retrieval accuracy. Our proposed method demonstrates enhancement in image retrieval speed when applied to the CLIP-based model, UNICOM, designed for category-level retrieval, as well as the CNN-based R-GeM model, tailored for particular object retrieval by validating its effectiveness across various domains and backbones. We achieve an 18-times speed improvement while preserving over 99% accuracy when applied to the In-Shop dataset, the largest dataset in the experiments.

An Image Retrieval Method for Lunar Complex Craters Integrating Visual and Depth Features

In the geological research of the Moon and other celestial bodies, the identification and analysis of impact craters are crucial for understanding the geological history of these bodies. With the rapid increase in the volume of high-resolution imagery data returned from exploration missions, traditional image retrieval methods face dual challenges of efficiency and accuracy when processing lunar complex crater image data. Deep learning techniques offer a potential solution. This paper proposes an image retrieval model for lunar complex craters that integrates visual and depth features (LC2R-Net) to overcome these difficulties. For depth feature extraction, we employ the Swin Transformer as the core architecture for feature extraction and enhance the recognition capability for key crater features by integrating the Convolutional Block Attention Module with Effective Channel Attention (CBAMwithECA). Furthermore, a triplet loss function is introduced to generate highly discriminative image embeddings, further optimizing the embedding space for similarity retrieval. In terms of visual feature extraction, we utilize Local Binary Patterns (LBP) and Hu moments to extract the texture and shape features of crater images. By performing a weighted fusion of these features and utilizing Principal Component Analysis (PCA) for dimensionality reduction, we effectively combine visual and depth features and optimize retrieval efficiency. Finally, cosine similarity is used to calculate the similarity between query images and images in the database, returning the most similar images as retrieval results. Validation experiments conducted on the lunar complex impact crater dataset constructed in this article demonstrate that LC2R-Net achieves a retrieval precision of 83.75%, showcasing superior efficiency. These experimental results confirm the advantages of LC2R-Net in handling the task of lunar complex impact crater image retrieval.

FFLDGA-Net: Image retrieval method based on Feature Fusion Learnable Descriptor Graph Attention Network

Image retrieval aims to retrieve and return the image in the database that is most similar to the query image. However, the performance of image retrieval models is often hindered by the limited dimensionality of images, which lacks depth information about objects. To address this issue, we propose a novel image retrieval model called FFLDGA-Net (Feature Fusion-based Learnable Descriptor Graph Attention Network). This model aims to overcome the absence of depth information in images by fusing feature information from both image data and point cloud data. First, we introduce the LDGA-Net, which effectively improved the model’s ability to mine hard samples and negative samples. Then, we combine the multi-scale route convolution module with a one-dimensional path aggregation network to fuse point clouds and image features at multiple scales, and establish a high-dimensional relationship between features and low-dimensional features. To mitigate training noise, we incorporate a soft label strategy tailored to the dataset’s characteristics. Our experimental results on two benchmark datasets demonstrate the significant improvements achieved by FFLDGA-Net in image retrieval performance.

Single-loss hash image retrieval method based on improved visual transformer

Deep hashing methods have gained popularity in image retrieval due to their advantages such as low storage requirements and high efficiency. However, existing deep hashing methods for large-scale image retrieval tasks suffer from issues including low discriminative power of binary hash codes, difficult optimization of losses, and low retrieval accuracy. This paper proposes a single-loss hash image retrieval method based on an improved visual transformer to address these issues. The proposed method utilizes a pre-trained Vision Transformer (ViT) on ImageNet as the backbone network, augmented with a hash coding layer to extract image features more comprehensively. Additionally, we design a single learning objective loss function that addresses the discriminative power of hash codes and quantization errors, thereby eliminating the complexity of adjusting various loss weights. Experimental evaluations on ImageNet100, NUS-WIDE, CIFAR10, and MS-COCO datasets demonstrate the superior performance of the proposed method compared to contemporary methods, indicating its adaptability to diverse data.

Incremental image retrieval method based on feature perception and deep hashing

Incremental image retrieval method based on feature perception and deep hashing

Research on unsupervised image retrieval methods based on contrastive learning

In the convergence of fashion and artificial intelligence (AI), significant strides have been made in areas such as clothing recognition, retrieval, and classification, enabled by advanced AI technologies and expansive annotated datasets. As the AI in Fashion market continues to surge, the future of the fashion industry promises to be redefined by intelligent, efficient, and more accessible solutions. Image retrieval, one of the important parts in AI, has experienced remarkable growth, empowered by advanced algorithms and vast annotated datasets, making it a crucial component in various domains such as digital libraries, online marketing. Therefore, this report mainly provides an extensive review of image retrieval methods and the emerging paradigm of contrastive learning, underscoring their relevance and applications in the realm of artificial intelligence. This paper primarily reviews the technologies in the amalgamation of the image retrieval field and contrastive learning. It elucidates the history and progression of image retrieval, offers a methodical analysis of the two primary approachestext-based image retrieval and content-based image retrievaland examines how contrastive learning is employed in image retrieval systems.