Multimodal Image Retrieval Research Articles

Learning to disentangle and fuse for fine-grained multi-modality ship image retrieval

Multi-modality ship image retrieval aims to retrieve ship images from a large dataset, encompassing various modalities, when provided with a query ship image. One of the key challenges is addressing intra-modality variations and cross-modality discrepancies resulting from complex image content and different types of imaging systems. Current multi-modality retrieval methods primarily focus on extracting heterogeneous features from diverse modalities and mapping them into a shared feature space. However, these methods fail to determine the meaningfulness of the extracted features for retrieval tasks, as the features extracted across different modalities exhibit highly complex interactions. To overcome this limitation, we propose a novel framework, dubbed Disentangled Fusion Network (DFN), to decompose ship images into attribute and identity features, capturing modality-specific details and facilitating ship identification. This is achieved through an image disentanglement and fusion module, separating features across different modalities and converting the challenging multi-modality retrieval problem into a more manageable single-modality problem. To enable fine-grained retrieval, a region mining module identifies discriminative areas within ship images. The training process utilizes a bilevel optimization strategy to learn parameters, optimizing both image fusion and retrieval. Extensive experiments conducted on a publicly available multi-modality ship image dataset demonstrate the superior performance of the proposed DFN in image fusion and retrieval tasks.

Multiple Pseudo-Siamese Network with Supervised Contrast Learning for Medical Multi-modal Retrieval

Medical multi-modal retrieval aims to provide doctors with similar medical images from different modalities, which can greatly promote the efficiency and accuracy of clinical diagnosis. However, most existing medical retrieval methods hardly support the retrieval of multi-modal medical images, i.e., the number of modalities is greater than 2, and just convert retrieval to classification or clustering. It futilely breaks the gap between the visual information and the semantic information in different medical image modalities. To solve the problem, a S upervised C ontrast L earning method based on a M ultiple P seudo- S iamese network (SCL-MPS) is proposed for multi-modal medical image retrieval. In order to make the samples with semantic similarity close neighbors on Riemann manifold, the multiple constraints based on semantic consistency and modal invariance are designed in different forward stages of SCL-MPS. We theoretically demonstrate the feasibility of the designed constraints. Finally, experiments on four benchmark datasets (ADNI1, ADNI2, ADNI3, and OASIS3) show that SCL-MPS achieves state-of-the-art performance compared to 15 retrieval methods. Especially, SCL-MPS achieves a 100% mAP score in medical cross-modal retrieval on ADNI1.

Privacy-Preserving Image Retrieval with Multi-Modal Query

Abstract The ever-growing multi-modal images pose great challenges to local image storage and retrieval systems. Cloud computing provides a solution to large-scale image data storage but suffers from privacy issues and lacks the support for multi-modal image retrieval. To address these, a searchable encryption-empowered privacy-preserving multi-modal image retrieval method is proposed. First, we design a hybrid image retrieval framework that fuses visual features and textual features at a decision level and further supports similar image retrieval and multi-keyword image retrieval. Second, we construct a new hybrid inverted index structure to distinguish high-frequency terms from low-frequency terms and index them through hierarchical index trees and data blocks, respectively, which greatly improves query efficiency. Third, we design a prime encoding-based multi-keyword query method that converts mapping operations in bloom filters into inner product calculations, and further implements secure multi-keyword image query. Experiments against the Baseline schemes are conducted to verify the performance of the scheme in terms of high efficiency.

Optimized transfer learning based multi-modal medical image retrieval

Optimized transfer learning based multi-modal medical image retrieval

Multimodal biomedical image retrieval and indexing system using handcrafted with deep convolution neural network features

The advances in biomedical imaging equipment have produced a massive amount of medical images that are generated by the different modalities. Consequently, a huge volume of data has been produced and caused a complex and time-consuming retrieving process of the relevant cases. To resolve this issue, the Content-Based Biomedical Image Retrieval (CBMIR) system is applied to retrieve the related images from the earlier patients’ databases. However, the previous handcrafted features methods that applied the CBMIR model have shown poor performance in many multimodal databases. In this paper, we focus on designing CBMIR technique using Deep Learning (DL) models. We present a new Multimodal Biomedical Image Retrieval and Classification (M-BMIRC) technique for retrieving and classifying the biomedical images from huge databases. The proposed M-BMIRC model involves three dissimilar processes as following: feature extraction, similarity measurement, and classification. It uses an ensemble of handcrafted features from Zernike Moments (ZM) and deep features from Deep Convolutional Neural Networks (DCNN) for feature extraction process. Additionally, the Hausdorff Distance based similarity measure is employed to identify the resemblance between the queried image and the images that exist in the database. Moreover, the classification process gets executed on the retrieval images using the Probabilistic Neural Network (PNN) model, which allocates the class labels of the tested images. Finally, the experimental studies are conducted using two benchmark medical datasets and the results ensure the superior performance of the proposed model in terms of different measures include Average Precision Rate (APR), Average Recall Rate (ARR), F-score, accuracy, and Computation Time (CT).

MRIS: A Multi-modal Retrieval Approach for Image Synthesis on Diverse Modalities.

Multiple imaging modalities are often used for disease diagnosis, prediction, or population-based analyses. However, not all modalities might be available due to cost, different study designs, or changes in imaging technology. If the differences between the types of imaging are small, data harmonization approaches can be used; for larger changes, direct image synthesis approaches have been explored. In this paper, we develop an approach based on multi-modal metric learning to synthesize images of diverse modalities. We use metric learning via multi-modal image retrieval, resulting in embeddings that can relate images of different modalities. Given a large image database, the learned image embeddings allow us to use k-nearest neighbor (k-NN) regression for image synthesis. Our driving medical problem is knee osteoarthritis (KOA), but our developed method is general after proper image alignment. We test our approach by synthesizing cartilage thickness maps obtained from 3D magnetic resonance (MR) images using 2D radiographs. Our experiments show that the proposed method outperforms direct image synthesis and that the synthesized thickness maps retain information relevant to downstream tasks such as progression prediction and Kellgren-Lawrence grading (KLG). Our results suggest that retrieval approaches can be used to obtain high-quality and meaningful image synthesis results given large image databases.

Multimodal Analogy-Based Image Retrieval by Improving Semantic Embeddings

In this work, we study the application of multimodal analogical reasoning to image retrieval. Multimodal analogy questions are given in a form of tuples of words and images, e.g., “cat”:“dog”::[an image of a cat sitting on a bench]:?, to search for an image of a dog sitting on a bench. Retrieving desired images given these tuples can be seen as a task of finding images whose relation between the query image is close to that of query words. One way to achieve the task is building a common vector space that exhibits analogical regularities. To learn such an embedding, we propose a quadruple neural network called multimodal siamese network. The network consists of recurrent neural networks and convolutional neural networks based on the siamese architecture. We also introduce an effective procedure to generate analogy examples from an image-caption dataset for training of our network. In our experiments, we test our model on analogy-based image retrieval tasks. The results show that our method outperforms the previous work in qualitative evaluation.

DenseBert4Ret: Deep bi-modal for image retrieval

In this study, we focus on the task of desired image retrieval from an extensive database. This comes under the purview of image retrieval systems. Users may input an image and a text as a query and expect the system to retrieve an image. The retrieved image should be close to the users’ desires expressed in the query. Nowadays, digital media generates more than petabytes of imagery data in one day. Moreover, due to the internet, this massive amount of data is readily available to users. However, extracting the desired images from such a colossal databank is a challenging task. Users always prefer to extract an image that reflects their wishful thinking, and the user may desire to alter their visionary thoughts according to their abstract thoughts. For example, Elizabeth wants to have a laptop that is similar to her friend’s laptop. However, she wants to have the same kind of laptop with a built-in GPU and in silver color. So she expects the e-business platform to show a laptop according to her wish. This paper attempts to devise a multi-modal algorithm for such tasks. It takes care of the user’s visual and textual query. It has a query image and text as input and retrieves an image similar to the input image but modified according to the text query. This study focuses on a multi-modal image retrieval system that processes both image and text as input queries. Users can input an image and a text query to modify the image or add more information on it. The text reflects the desired modifications in the image. We proposed a bi-modal image retrieval system named “DenseBert4Ret” that learns image and text features concurrently. As the name indicates, DenseNet and BERT models are used for image and text features extractions, respectively. It is based on deep learning techniques used for the joint representation of image and text features. We trained the model, which forces the input image to be modified according to the user’s textual query. We used deep information learning to train and test our model on three challenging real-world datasets, i.e., MIT States, Fshion200K and FashionIQ. We also show that the proposed model outperforms its predecessor with tuned parameters.

Improved Content Based Image Retrieval Process Based on Deep Convolutional Neural Network and Salp Swarm Algorithm

Digital image and medical image retrieval from several repositories are improving gradually, so the capacity of repositories increases rapidly. The semantic space is the main issue on content-based image retrieval (CBIR), which exists among the semantic level as well as increases the data recognized through human and low level visible data obtained through the image. The CBIR system utilizes the deep convolutional neural network (DCNN), which is trained to medical image characterization and the digital image by salp swarm optimization algorithm (SSA). The average classification accuracy for medical image is 86.805%, a mean average precision is 79%, Average Recall Rate (ARR) is 91.7% and [Formula: see text]-measure is 84.9%, are achieved during retrieval task. For image retrieval, the Average Precision Rate (APR) improved from 39%, 40%, 36% and 42.5% to 86.8% and the ARR enhanced from 39.5%, 40.5%, 35.5% and 42.5% to 86.8%. The [Formula: see text]-measure is improved from 39.5%, 40.5%, 35.5% and 42.5% to 86.8% as different with Local tetra patterns (LTrP), LOOP, local derivative pattern (LDP) and local mean differential excitation pattern (LMDeP) separately on Corel-1K dataset. The presented method is most suitable for multimodal digital images and medical image retrieval for various parts of the body.

BigEarthNet-MM: A Large-Scale, Multimodal, Multilabel Benchmark Archive for Remote Sensing Image Classification and Retrieval [Software and Data Sets

This paper presents the multi-modal BigEarthNet (BigEarthNet-MM) benchmark archive made up of 590,326 pairs of Sentinel-1 and Sentinel-2 image patches to support the deep learning (DL) studies in multi-modal multi-label remote sensing (RS) image retrieval and classification. Each pair of patches in BigEarthNet-MM is annotated with multi-labels provided by the CORINE Land Cover (CLC) map of 2018 based on its thematically most detailed Level-3 class nomenclature. Our initial research demonstrates that some CLC classes are challenging to be accurately described by only considering (single-date) BigEarthNet-MM images. In this paper, we also introduce an alternative class-nomenclature as an evolution of the original CLC labels to address this problem. This is achieved by interpreting and arranging the CLC Level-3 nomenclature based on the properties of BigEarthNet-MM images in a new nomenclature of 19 classes. In our experiments, we show the potential of BigEarthNet-MM for multi-modal multi-label image retrieval and classification problems by considering several state-of-the-art DL models. We also demonstrate that the DL models trained from scratch on BigEarthNet-MM outperform those pre-trained on ImageNet, especially in relation to some complex classes, including agriculture and other vegetated and natural environments. We make all the data and the DL models publicly available at https://bigearth.net, offering an important resource to support studies on multi-modal image scene classification and retrieval problems in RS.

Medical image retrieval for detecting pneumonia using binary classification with deep convolutional neural networks

In the concepts of Deep Learning, the ConvNets are a classified group of deep neural networks that are used to analyze the images. They find their major applications in image classification and in the medical image analysis. ConvNets are regularized, multilayer preceptors. The major challenge today is the increasing size of medical image repositories which is leading to troubles in large database management and then querying those databases for storage and retrieval and this can only be sorted out using the Content Based Medical Image Retrieval (CBMIR) systems. A major challenge in CBMIR systems is the semantic gap that exists between the low-level visual information captured by the imaging devices and high-level semantic information captured by human. This paper demonstrates a new and efficient framework with the algorithm of Deep CNN for the Feature Based Medical Image Retrieval (FBMIR) system for fast and efficient retrieval of medical and clinical images for detecting Pneumonia. The intermodal datasets are used and divided into two classes to train the network. One class consists of the lungs infected by Pneumonia and the other class contains the images of lungs which are normal. The learned features from the trained deep convolutional neural network with the fast processing feature of the hash method are used to reduce the feature space. To achieve the best retrieval results with class-based predictions 93.9% of average classification accuracy is achieved and 0.87 of mean average precision is achieved for this retrieval task. The proposed framework fit the best for top-ranked multimodal medical image retrieval.

Benchmarking Image Retrieval Diversification Techniques for Social Media

Image retrieval has been an active research domain for over 30 years and historically it has focused primarily on precision as an evaluation criterion. Similar to text retrieval, where the number of indexed documents became large and many relevant documents exist, it is of high importance to highlight diversity in the search results to provide better results for the user. The Retrieving Diverse Social Images Task of the MediaEval benchmarking campaign has addressed exactly this challenge of retrieving diverse and relevant results for the past years, specifically in the social media context. Multimodal data (e.g., images, text) was made available to the participants including metadata assigned to the images, user IDs, and precomputed visual and text descriptors. Many teams have participated in the task over the years. The large number of publications employing the data and also citations of the overview articles underline the importance of this topic. In this paper, we introduce these publicly available data resources as well as the evaluation framework, and provide an in-depth analysis of the crucial aspects of social image search diversification, such as the capabilities and the evolution of existing systems. These evaluation resources will help researchers for the coming years in analyzing aspects of multimodal image retrieval and diversity of the search results.

Diversity-based interactive learning meets multimodality

In interactive retrieval tasks, one of the main objectives is to maximize the user information gain throughout search sessions. Retrieving many relevant items is quite important, but it does not necessarily completely satisfy the user needs. When only relevant near-duplicate items are retrieved, the amount of different concepts users are able to extract from the target collection is very limited. Therefore, broadening the number of concepts present in a result set may improve the overall search experience. Diversifying concepts present in the retrieved set is one possibility for increasing the information gain in a single search iteration, maximizing the likelihood of including at least some relevant items for each possible intent of ambiguous or underspecified queries. Relevance feedback approaches may also take advantage of diverse results to improve internal machine learning models. In this context, this work proposes and analyses several multimodal image retrieval approaches built over a learning framework for relevance feedback on diversified results. Our experimental analysis shows that different retrieval modalities are positively impacted by diversity, but achieve best retrieval effectiveness with diversification applied at different moments of a search session. Moreover, the best results are achieved with a query-by-example approach using multimodal information obtained from feedback. In summary, we demonstrate that learning with diversity is an effective alternative for boosting multimodal interactive learning approaches.

Measuring and Predicting Tag Importance for Image Retrieval.

Textual data such as tags, sentence descriptions are combined with visual cues to reduce the semantic gap for image retrieval applications in today's Multimodal Image Retrieval (MIR) systems. However, all tags are treated as equally important in these systems, which may result in misalignment between visual and textual modalities during MIR training. This will further lead to degenerated retrieval performance at query time. To address this issue, we investigate the problem of tag importance prediction, where the goal is to automatically predict the tag importance and use it in image retrieval. To achieve this, we first propose a method to measure the relative importance of object and scene tags from image sentence descriptions. Using this as the ground truth, we present a tag importance prediction model to jointly exploit visual, semantic and context cues. The Structural Support Vector Machine (SSVM) formulation is adopted to ensure efficient training of the prediction model. Then, the Canonical Correlation Analysis (CCA) is employed to learn the relation between the image visual feature and tag importance to obtain robust retrieval performance. Experimental results on three real-world datasets show a significant performance improvement of the proposed MIR with Tag Importance Prediction (MIR/TIP) system over other MIR systems.

AN OBJECT PROPERTIES FILTER FOR MULTI-MODALITY ONTOLOGY SEMANTIC IMAGE RETRIEVAL

Ontology is a semantic technology that provides the possible approach to bridge the issue on semantic gap in image retrieval between low-level visual features and high-level human semantic. The semantic gap occurs when there is a discrepancy between the information that is extracted from visual data and the text description. In other words, there is a difference between the computational representation in machine and human natural language. In this paper, an ontology has been utilized to reduce the semantic gap by developing a multi-modality ontology image retrieval with the enhancement of a retrieval mechanism by using the object properties filter. To achieve this, a multi-modality ontology semantic image framework was proposed, comprising of four main components which were resource identification, information extraction, knowledge-based construction and retrieval mechanism. A new approach, namely object properties filter is proposed by customizing the semantic image retrieval algorithm and the graphical user interface to facilitate the user to engage with the machine i.e. computers, in order to enhance the retrieval performance. The experiment results showed that the proposed approach delivered better results compared to the approach that did not use the object properties filter based on probability precision measurement.

A Multimodal Search Engine for Medical Imaging Studies.

The use of digital medical imaging systems in healthcare institutions has increased significantly, and the large amounts of data in these systems have led to the conception of powerful support tools: recent studies on content-based image retrieval (CBIR) and multimodal information retrieval in the field hold great potential in decision support, as well as for addressing multiple challenges in healthcare systems, such as computer-aided diagnosis (CAD). However, the subject is still under heavy research, and very few solutions have become part of Picture Archiving and Communication Systems (PACS) in hospitals and clinics. This paper proposes an extensible platform for multimodal medical image retrieval, integrated in an open-source PACS software with profile-based CBIR capabilities. In this article, we detail a technical approach to the problem by describing its main architecture and each sub-component, as well as the available web interfaces and the multimodal query techniques applied. Finally, we assess our implementation of the engine with computational performance benchmarks.

Multimodal medical image retrieval system

In this paper we depict an implemented system for medical image retrieval. Our system performs retrieval based on both textual and visual content, separately and combined, using advanced encoding and quantization techniques. The text-based retrieval subsystem uses textual data acquired from an image's corresponding article to generate a suitable representation. Using a vector space model, the generated representations structure is altered to increase performance. Query expansion with pseudo-relevance feedback is applied to fine-tune the results. The content-based retrieval subsystem performs retrieval based on visual features extracted from the images. A Gaussian Mixture Model is constructed from the extracted visual features, in our case - RGB histograms, and is used in encoding the same features into Fisher Vectors. With scalability and speed in mind, we utilized a product quantization technique over the generated vectors, which provides fast response times over large image collections. Product quantization drastically reduces the size of the image representation at almost no cost to accuracy, thus improving the scalability factor of our system. Our system uses modality classification to further improve retrieval results. This subsystem labels the image modality based on their visual content. The images are described using state-of-the-art opponentSIFT visual features. Classification was performed using Support Vector Machines (SVMs). The predictions from the SVMs are used for re-ranking the resulting images based on their modality and the modality of the query. The system was evaluated against the standardized ImageCLEF 2013, 2012 and 2011 medical datasets and it reported state-of-the-art performance for all datasets.

Multimodal Image Retrieval Based on Query Weighting

Image retrieval methods try to increase the performance of retrieval by adjusting weights of relevant and non-relevant images. Adjusting weights of queries instead of images is studied in this paper to see whether it increases or decreases performance of retrieval system. Evaluation measure of the method is calculated for each query instead of each sample. In proposed method, textual and visual features of images are used to retrieve ranked results. To train visual module, query weighting and updating weights in each iteration are used to concentrate on the queries which did not provide acceptable results and also cause performance drop of final retrieval. Compared with other previous researches, the results of applying the proposed method to IAPR TC-12 dataset indicate high efficiency of proposed method. Image retrieval methods like this can be used in a lot of applications such as, gasometer, digit recognition using gasometer, digits image etc..

Keyword Visual Representation for Image Retrieval and Image Annotation

Keyword-based image retrieval is more comfortable for users than content-based image retrieval. Because of the lack of semantic description of images, image annotation is often used a priori by learning the association between the semantic concepts (keywords) and the images (or image regions). This association issue is particularly difficult but interesting because it can be used for annotating images but also for multimodal image retrieval. However, most of the association models are unidirectional, from image to keywords. In addition to that, existing models rely on a fixed image database and prior knowledge. In this paper, we propose an original association model, which provides image-keyword bidirectional transformation. Based on the state-of-the-art Bag of Words model dealing with image representation, including a strategy of interactive incremental learning, our model works well with a zero-or-weak-knowledge image database and evolving from it. Some objective quantitative and qualitative evaluations of the model are proposed, in order to highlight the relevance of the method.

A semi-supervised learning algorithm for relevance feedback and collaborative image retrieval

Abstract The interaction of users with search services has been recognized as an important mechanism for expressing and handling user information needs. One traditional approach for supporting such interactive search relies on exploiting relevance feedbacks (RF) in the searching process. For large-scale multimedia collections, however, the user efforts required in RF search sessions is considerable. In this paper, we address this issue by proposing a novel semi-supervised approach for implementing RF-based search services. In our approach, supervised learning is performed taking advantage of relevance labels provided by users. Later, an unsupervised learning step is performed with the objective of extracting useful information from the intrinsic dataset structure. Furthermore, our hybrid learning approach considers feedbacks of different users, in collaborative image retrieval (CIR) scenarios. In these scenarios, the relationships among the feedbacks provided by different users are exploited, further reducing the collective efforts. Conducted experiments involving shape, color, and texture datasets demonstrate the effectiveness of the proposed approach. Similar results are also observed in experiments considering multimodal image retrieval tasks.