Image Retrieval Framework Research Articles

SMedIR: secure medical image retrieval framework with ConvNeXt-based indexing and searchable encryption in the cloud

The security and privacy of medical images are crucial due to their sensitive nature and the potential for severe consequences from unauthorized modifications, including data breaches and inaccurate diagnoses. This paper introduces a method for lossless medical image retrieval from encrypted images stored on third-party clouds. The proposed approach employs a symmetric integrity-centric image encryption scheme, leveraging multiple chaotic maps and cryptographic hash techniques, to ensure lossless image reconstruction. Medical images are first encrypted by the image owners and converted into hashcodes encapsulating essential features using a deep hashing technique with the ConvNeXt network as the backbone in parallel. To ensure index privacy, these hashcodes are encrypted in a searchable manner. The encrypted medical images, along with a secure index, are subsequently uploaded to cloud storage. Authorized medical image users can request similar medical images for diagnostic purposes by submitting a query image, from which a search trapdoor is generated and sent to the cloud. The retrieval process involves a secure similar image search over the encrypted indexes, followed by decryption along with integrity verification of the retrieved images. The proposed method has been rigorously tested on three standard medical datasets, demonstrating an improvement of 5-20% in retrieval accuracy compared to standard baselines. Formal security analysis and experimental results indicate that the proposed scheme offers enhanced security and retrieval accuracy, making it an effective solution for the encrypted storage and secure retrieval of medical image data.

An integrated blockchain and fractional DCT based highly secured framework for storage and retrieval of retinal images

The digital image capability of storing large amount of data has resulted in it’s increased popularity. Images are used to transmit large amount of information across different geographical locations using different cloud services. Securing these digitally stored images has remained a challenging task for researchers as they are prone to cyber- attacks. A potential solution to this problem can be blockchain, which provide secure and unchangeable storage. However, securing images on blockchain has another challenge as the image size increases the associated cost involved in blockchain. Fractional Discrete Cosine Transform(fctDCT) has the capability to minimizes the amount of data necessary for expressing an image in a secure way. This paper presents a novel framework for securely storing and retrieving medical images by extracting feature maps from medical images by fctDCT, followed by encoding and storing the feature map on decentralized cloud and linking them on blockchain. The integration has been implemented by using four different α angles which are stored on blockchain and are needed to be same at storage and retrieval stage as only the authentic user would have access to unique α angles and number of coefficients that have been used in storing their medical images. The performance of proposed framework has been evaluated by employing image quality metric such as Mean Square Error (MSE), Peak Signal to Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) and multi-SSIM by comparing it with correct and incorrect α values on four different values of α.

A Framework for Extensive Content-Based Image Retrieval System Incorporating Relevance Feedback and Query Suggestion

Over the last decade, there is a huge amount of images in which visual information has become increasingly important. If the images are to be utilized, they must be organized into databases, from there it can be searched based on different criteria. Content-based Image retrieval (CBIR) frameworks have turned into a mainstream subject of exploration; for their ability to retrieve images based on the real visual content rather than by manually connected textual descriptions. In the CBIR framework, analysis and interpretation of image information in large and diverse image databases is evidently complex because there is no prior information on the size or scale of individual structures within the images to be analyzed. In CBIR, retrieval is based on visual image features, which can be extracted automatically from the images with the help of human intervention, namely a technique called relevance feedback. Nonetheless, an efficient way of differentiating the visual content of images is complex to produce. Therefore, rather than a perfect solution CBIR systems must be able to exploit a partial solution to the problem of image understanding. In this paper, there is implementation of a CBIR framework is introduced that not only tries to efficiently capture the user intent based on the feedback but also provide query suggestions that can help its users to pose better queries in order to retrieve desired results in an efficient manner. The proposed technique is straightforward to implement and scopes efficiently to huge datasets. Extensive experiments on diverse real datasets with image similarity measures have revealed the dominance of the proposed method over original algorithms.

Vision-language joint representation learning for sketch less facial image retrieval

The traditional sketch-based facial image retrieval (SBFIR) framework assumes that a high-quality facial sketch has been prepared prior to the retrieval task. However, drawing such a sketch requires considerable skills and is time consuming, resulting in limited applicability. Sketch less facial image retrieval (SLFIR) framework aims to break these barriers through human–computer interaction during the sketching process. The primary challenges for the SLFIR problem can be noted that initial sketches (at early sketching) contain only local details and exhibit significant differences among users, resulting in poor performance at early stages and weak generalization abilities in practical testing. In this study, we developed a vision–language pretraining model to align the representation of facial images and their associated semantics. Based on this framework, we proposed a method for learning joint representations by fusing sketches with prior semantics, thereby enriching the information of initial sketches. Specifically, (1) we developed a series of well-designed operations to improve the quality of facial image–text pairs in the LAION-Face dataset; we trained a facial vision–language pretraining (FVLP) model to align the facial image and its semantics at the feature level. (2) subsequently, using FVLP as the backbone, we designed a convolutional attention module to fuse the multiscale features extracted from image encoder. This facilitated the learning of a multimodal representation crucial for the final retrieval process. In experiments, our proposed method achieved state-of-the-art performance at early stages on two public datasets; moreover, it exhibited good generalization capabilities during practical testing. Thus, our method significantly outperforms other baselines in terms of early retrieval performance. Codes are available at: https://github.com/ddw2AIGROUP2CQUPT/FVLP.

Prior semantic-embedding representation learning for on-the-fly FG-SBIR

In fine-grained sketch-based image retrieval (FG-SBIR) framework, sketch drawing is a time-consuming and skill-intensive process that greatly restricts its practical application. Our study aims to enhance the early-stage retrieval efficiency by utilizing partial sketches with a minimum number of strokes in the on-the-fly FG-SBIR framework. Nevertheless, partial sketches with few strokes usually lack global information and consist only of sparse local details. This lack of detail results in ambiguous semantic content and indistinct differences among images, which consequently impairs early-stage retrieval. To acquire an efficient representation for these sketches, we propose a multi-granularity feature representation model that utilizes prior knowledge embedding to deal with the sketch drawing process. In the first stage,we design a triplet network to learn the joint a coarse representation embedding space that is common between the complete sketch and the image. In the second stage, We enhanced the semantic representation of sketches by utilizing prior knowledge, which aids in supplementing missing fine-grained features. This was achieved by formalizing the prior knowledge and integrating it into the feature vector obtained in the initial stage, thereby acquiring the final representation of a subset of sketches. Experiments conducted on two public datasets indicate that our method outperforms the state-of-the-art baseline during the early retrieval phase. In practical applications, our approach has also achieved good results.

VTHSC-MIR: Vision Transformer Hashing with Supervised Contrastive learning based medical image retrieval

In past few years, deep learning based medical image analysis technologies have significantly improved computer-assisted tasks like detecting, diagnosing, and predicting medical outcomes. The monitoring and diagnosis of ailments such as cancer and COVID-19 rely primarily on retrieving medical images. As the medical databases size is increasing rapidly it causes difficulty in managing and querying them. The main challenge lies in achieving superior retrieval performance while handling the intricacies of medical imaging datasets. To tackle this, we propose a novel end-to-end trainable framework for medical image retrieval using a vision transformer hashing with Supervised Contrastive (VTHSC) learning. Leveraging the self-attention mechanism of transformers and supervised contrastive loss, our model surpasses existing hashing-based retrieval methods. Focusing on the most recent COVID-19 and breast cancer datasets, we aim to efficiently retrieve relevant medical images from large datasets. Using the ImageNet-pretrained ViT as its backbone network, the VTHSC model incorporates a hashing head combined with a supervised contrastive loss and employs joint loss optimization. The VTHSC model is subsequently fine- tuned for a retrieval task, employing four different hashing frameworks: Deep Supervised Hashing (DSH), GreedyHash, Improved Deep Hashing Network (IDHN), and Deep Polarized Network (DPN). Our model achieves outstanding Mean Average Precision (MAP) scores of 98.9 for the BreakHis dataset and 96.03 for the COVID dataset. Notably, the VTHSC model surpasses several competing hashing-based retrieval methods by a substantial gain in terms of performance across various metrics such as precision, recall, and Top-6 retrieved images retrieval performance on the two benchmark medical image datasets.

Triplet-constrained deep hashing for chest X-ray image retrieval in COVID-19 assessment

Radiology images of the chest, such as computer tomography scans and X-rays, have been prominently used in computer-aided COVID-19 analysis. Learning-based radiology image retrieval has attracted increasing attention recently, which generally involves image feature extraction and finding matches in extensive image databases based on query images. Many deep hashing methods have been developed for chest radiology image search due to the high efficiency of retrieval using hash codes. However, they often overlook the complex triple associations between images; that is, images belonging to the same category tend to share similar characteristics and vice versa. To this end, we develop a triplet-constrained deep hashing (TCDH) framework for chest radiology image retrieval to facilitate automated analysis of COVID-19. The TCDH consists of two phases, including (a) feature extraction and (b) image retrieval. For feature extraction, we have introduced a triplet constraint and an image reconstruction task to enhance discriminative ability of learned features, and these features are then converted into binary hash codes to capture semantic information. Specifically, the triplet constraint is designed to pull closer samples within the same category and push apart samples from different categories. Additionally, an auxiliary image reconstruction task is employed during feature extraction to help effectively capture anatomical structures of images. For image retrieval, we utilize learned hash codes to conduct searches for medical images. Extensive experiments on 30,386 chest X-ray images demonstrate the superiority of the proposed method over several state-of-the-art approaches in automated image search. The code is now available online.

A novel blockchain and fractional DCT based image storage and retrieval framework

Securing medical images are a challenging task for researchers as they are prone to cyber- attacks. A potential solution to this problem can be blockchain, which provide secure and unchangeable storage. Fractional Discrete Cosine Transform (frDCT) has the capability to minimizes the amount of data necessary for expressing an image in a secure way. This paper presents a novel framework for compressing as well as securely storing and retrieving medical images by extraction of feature maps from medical images by frDCT, followed by encoding and storing the feature map on decentralized cloud and linking them on blockchain using fractional angle α. The proposed framework provides numerous benefits such as improved data sharing and collaboration, enhanced security, compression as well as efficient retrieval and processing of medical image data which in turn decreases the computational energy requirement providing sustainability. The performance of proposed framework has been evaluated by employing image quality metric such as MSE, PSNR, SSIM and multi-SSIM by comparing it with correct and incorrect α values on six different values of frDCT parameter α.

Dual Modality Reverse Re-Ranking (DM-RR) based image retrieval framework

Dual Modality Reverse Re-Ranking (DM-RR) based image retrieval framework

Efficient Retrieval of Images with Irregular Patterns Using Morphological Image Analysis: Applications to Industrial and Healthcare Datasets.

Image retrieval is the process of searching and retrieving images from a datastore based on their visual content and features. Recently, much attention has been directed towards the retrieval of irregular patterns within industrial or healthcare images by extracting features from the images, such as deep features, colour-based features, shape-based features, and local features. This has applications across a spectrum of industries, including fault inspection, disease diagnosis, and maintenance prediction. This paper proposes an image retrieval framework to search for images containing similar irregular patterns by extracting a set of morphological features (DefChars) from images. The datasets employed in this paper contain wind turbine blade images with defects, chest computerised tomography scans with COVID-19 infections, heatsink images with defects, and lake ice images. The proposed framework was evaluated with different feature extraction methods (DefChars, resized raw image, local binary pattern, and scale-invariant feature transforms) and distance metrics to determine the most efficient parameters in terms of retrieval performance across datasets. The retrieval results show that the proposed framework using the DefChars and the Manhattan distance metric achieves a mean average precision of 80% and a low standard deviation of ±0.09 across classes of irregular patterns, outperforming alternative feature-metric combinations across all datasets. Our proposed ImR framework performed better (by 8.71%) than Super Global, a state-of-the-art deep-learning-based image retrieval approach across all datasets.

Privacy-Preserving Image Retrieval with Multi-Modal Query

Abstract The ever-growing multi-modal images pose great challenges to local image storage and retrieval systems. Cloud computing provides a solution to large-scale image data storage but suffers from privacy issues and lacks the support for multi-modal image retrieval. To address these, a searchable encryption-empowered privacy-preserving multi-modal image retrieval method is proposed. First, we design a hybrid image retrieval framework that fuses visual features and textual features at a decision level and further supports similar image retrieval and multi-keyword image retrieval. Second, we construct a new hybrid inverted index structure to distinguish high-frequency terms from low-frequency terms and index them through hierarchical index trees and data blocks, respectively, which greatly improves query efficiency. Third, we design a prime encoding-based multi-keyword query method that converts mapping operations in bloom filters into inner product calculations, and further implements secure multi-keyword image query. Experiments against the Baseline schemes are conducted to verify the performance of the scheme in terms of high efficiency.

Deep-view linguistic and inductive learning (DvLIL) based framework for Image Retrieval

Deep-view linguistic and inductive learning (DvLIL) based framework for Image Retrieval

SE‐Swin: An improved Swin‐Transfomer network of self‐ensemble feature extraction framework for image retrieval

AbstractThe Swin‐Transformer is a variant of the Vision Transformer, which constructs a hierarchical Transformer that computes representations with shifted windows and window multi‐head self‐attention. This method can handle the scale invariance problem and performs well in many computer vision tasks. In image retrieval, high‐quality feature descriptors are necessary to improve retrieval accuracy. This paper proposes a self‐ensemble Swin‐Transformer network structure to fuse the features of different layers of the Swin‐Transformer network, eliminating noise points present in a single layer, and improving the retrieval effect. Two experiments were conducted, one on the In‐shop Clothes Retrieval dataset and another on the Stanford Online Product dataset. The experiments showed that the proposed method significantly increased the retrieval effect of features extracted using Vision Transformer, surpassing previous state‐of‐the‐art image retrieval methods. In the second experiment, the feature map of the trained model was visualized, revealing that the improved network significantly reduces focus on some noise points and enhances focus on image features compared to the original network.

Developing a model semantic‐based image retrieval by combining KD‐Tree structure with ontology

AbstractThe paper proposes an alternative approach to improve the performance of image retrieval. In this work, a framework for image retrieval based on machine learning and semantic retrieval is proposed. In the preprocessing phase, the image is segmented objects by using Graph‐cut, and the feature vectors of objects presented in the image and their visual relationships are extracted using R‐CNN. The feature vectors, visual relationships, and their symbolic labels are stored in KD‐Tree data structures which can be used to predict the label of objects and visual relationships later. To facilitate semantic query, the images use the RDF data model and create an ontology for the symbolic labels annotated. For each query image, after extracting their feature vectors, the KD‐Tree is used to classify the objects and predict their relationship. After that, a SPARQL query is built to extract a set of similar images. The SPARQL query consists of triple statements describing the objects and their relationship which were previously predicted. The evaluation of the framework with the MS‐COCO dataset and Flickr showed that the precision achieved scores of 0.9218 and 0.9370, respectively.

A hierarchical approach based CBIR scheme using shape, texture, and color for accelerating retrieval process

In the traditional content-based image retrieval (CBIR) framework, images are retrieved based on the combined primitive features. However, such a kind of fusion is always not effective as one feature may overshadow other image feature. To overcome this issue, in this particular paper, we have suggested a hierarchical framework where features like color, texture, and shape are considered in a single hierarchy only. The main concern of the hierarchical system is the proper selection of the order of retrieving visual features. So, semantic-based image retrieval has been carried out. Here, at the first level, the salience map-based region of interest has been identified, and then edge histogram descriptor-based shape features are incorporated. In the second hierarchy, we have proposed a novel directional texture feature extraction based on the Tamura features’ directionality. Further, color is considered another primitive feature of an image, but human visual perception is not sensitive to each color. The image can be visualized by the salient colors, and in this work, we have developed a color image quantization-based approach. Now, to validate the system, extensive experimental results and its comparison with its contemporaries through Corel-1 K, GHIM-10 K, Olivia-2688, and Produce-1400 databases have been carried out.

Exploring multivariate generalized gamma manifold for color texture retrieval

This work proposes a novel method for color-textured image retrieval on a Multivariate Generalized Gamma Distribution manifold (MGΓD). Thanks to the Gaussian copula theory, we define the expression of MGΓD, which efficiently models the statistical dependence structure between dual-tree complex wavelet transform (DTCWT) of the color components. The major contribution of this paper is to provide a geometric perspective to the MGΓD by treating it as a Riemannian manifold while proposing the geodesic distance (GD) as a measure of Riemannian similarity on it. Based on information geometry tools, we conduct a geometrical study of the MGΓD manifold, allowing us to derive two suitable approximations of the GD. The experiments are performed on five well-known color texture databases, considering the content-based image retrieval (CBIR) framework and using the RGB color space. The obtained results demonstrate the efficiency of the geometric interpretation through the proposed GD as a natural and intuitive similarity measure on the studied statistical manifold.

Learning relation-based features for fine-grained image retrieval

Fine-Grained Image Retrieval (FGIR) is a fundamental yet challenging task that has recently received considerable attention. However, two critical issues remain unresolved. On the one hand, convolutional neural networks (CNNs) trained with image-level labels tend to focus on the most discriminative image patches but overlook the implicit relation among them. On the other hand, existing large models developed for FGIR are computationally expensive and difficult to learn discriminative features. To address these issues without additional object-level annotations or localization sub-networks, we propose a novel unified framework for fine-grained image retrieval. Specifically, a novel Relation-based Convolutional Descriptor Aggregation (RCDA) method for extracting subtle yet discriminative features from fine-grained images is introduced. The RCDA method consists of a local feature generation network and a relation extraction (RE) module that models both explicit information and implicit relations. The explicit information is modeled by computing feature similarities, while the implicit relation is mined via an expectation-maximization algorithm. Moreover, we further leverage the knowledge distillation technique to optimize the parameters of the feature generation network and speed up the fine-tuning procedure by transferring knowledge from a large model to a smaller model. Experimental results on three benchmark datasets (CUB-200-2011, Stanford-Car and FGVC-Aircraft) demonstrate that the proposed method not only achieves a significant improvement over baseline models but also outperforms state-of-the-art methods by a large margin (6.4%, 1.3%, 23.2%, respectively).

Editorial Issue 34.2

Editorial Issue 34.2

RetCCL: Clustering-guided contrastive learning for whole-slide image retrieval.

Benefiting from the large-scale archiving of digitized whole-slide images (WSIs), computer-aided diagnosis has been well developed to assist pathologists in decision-making. Content-based WSI retrieval can be a new approach to find highly correlated WSIs in a historically diagnosed WSI archive, which has the potential usages for assisted clinical diagnosis, medical research, and trainee education. During WSI retrieval, it is particularly challenging to encode the semantic content of histopathological images and to measure the similarity between images for interpretable results due to the gigapixel size of WSIs. In this work, we propose a Retrieval with Clustering-guided Contrastive Learning (RetCCL) framework for robust and accurate WSI-level image retrieval, which integrates a novel self-supervised feature learning method and a global ranking and aggregation algorithm for much improved performance. The proposed feature learning method makes use of existing large-scale unlabeled histopathological image data, which helps learn universal features that could be used directly for subsequent WSI retrieval tasks without extra fine-tuning. The proposed WSI retrieval method not only returns a set of WSIs similar to a query WSI, but also highlights patches or sub-regions of each WSI that share high similarity with patches of the query WSI, which helps pathologists interpret the searching results. Our WSI retrieval framework has been evaluated on the tasks of anatomical site retrieval and cancer subtype retrieval using over 22,000 slides, and the performance exceeds other state-of-the-art methods significantly (around 10% for the anatomic site retrieval in terms of average mMV@10). Besides, the patch retrieval using our learned feature representation offers a performance improvement of 24% on the TissueNet dataset in terms of mMV@5 compared with using ImageNet pre-trained features, which further demonstrates the effectiveness of the proposed CCL feature learning method.

Ensemble learning framework for image retrieval via deep hash ranking

Deep hashing combines feature extraction or representation with hash coding jointly, which can significantly improve the speed of large-scale image retrieval. However, we notice that compared with traditional retrieval methods, due to the reduction of dimension and information loss, the retrieval performance of binaryhash coding has declined to a certain extent. Most hash retrieval algorithms focus on the semantic similarity between image pairs, and ignore the ranking information between the returned samples. The returned samples should not only match the retrieved samples, but also rank the correct samples in front of the returned list. In addition, the performance difference of the deep model used in deep hash retrieval will also limit the efficiency of retrieval. To address such problem, we proposed an ensemble deep neural model robust framework for image retrieval, which can learn compact hash codes containing rich semantic information through hash constraints. The ensemble strategy is introduced, and the weighted voting is applied to integrate the ranking list. Comprehensive experiments on three benchmark datasets show that the proposed method achieves very competitive results. Codes are available at https://github.com/lidonggen-123/Ensemble_Deephash_Image_Retrieval.