Cross-modal Information Retrieval Research Articles

VITR: Augmenting Vision Transformers with Relation-Focused Learning for Cross-Modal Information Retrieval

The relations expressed in user queries are vital for cross-modal information retrieval. Relation-focused cross-modal retrieval aims to retrieve information that corresponds to these relations, enabling effective retrieval across different modalities. Pre-trained networks, such as Contrastive Language-Image Pre-training networks, have gained significant attention and acclaim for their exceptional performance in various cross-modal learning tasks. However, the Vision Transformer (ViT) used in these networks is limited in its ability to focus on image region relations. Specifically, ViT is trained to match images with relevant descriptions at the global level, without considering the alignment between image regions and descriptions. This article introduces VITR, a novel network that enhances ViT by extracting and reasoning about image region relations based on a local encoder. VITR is comprised of two key components. Firstly, it extends the capabilities of ViT-based cross-modal networks by enabling them to extract and reason with region relations present in images. Secondly, VITR incorporates a fusion module that combines the reasoned results with global knowledge to predict similarity scores between images and descriptions. The proposed VITR network was evaluated through experiments on the tasks of relation-focused cross-modal information retrieval. The results derived from the analysis of the Flickr30K, MS-COCO, RefCOCOg, and CLEVR datasets demonstrated that the proposed VITR network consistently outperforms state-of-the-art networks in image-to-text and text-to-image retrieval.

RoCC:robust covert communication based on cross-modal information retrieval

RoCC:robust covert communication based on cross-modal information retrieval

Multi-Modal Machine Learning in Engineering Design: A Review and Future Directions

Abstract In the rapidly advancing field of multi-modal machine learning (MMML), the convergence of multiple data modalities has the potential to reshape various applications. This paper presents a comprehensive overview of the current state, advancements, and challenges of MMML within the sphere of engineering design. The review begins with a deep dive into five fundamental concepts of MMML: multi-modal information representation, fusion, alignment, translation, and co-learning. Following this, we explore the cutting-edge applications of MMML, placing a particular emphasis on tasks pertinent to engineering design, such as cross-modal synthesis, multi-modal prediction, and cross-modal information retrieval. Through this comprehensive overview, we highlight the inherent challenges in adopting MMML in engineering design, and proffer potential directions for future research. To spur on the continued evolution of MMML in engineering design, we advocate for concentrated efforts to construct extensive multi-modal design datasets, develop effective data-driven MMML techniques tailored to design applications, and enhance the scalability and interpretability of MMML models. MMML models, as the next generation of intelligent design tools, hold a promising future to impact how products are designed.

LANGUAGE DETECTION USING MACHINE LEARNING

This paper presents an innovative Machine Learning (ML) model for language detection that combines the power of logistic regression with a multimodal approach. The proposed model is designed to handle three types of inputs: sequential text data, files, and image representations. The proposed model offers a versatile and accurate solution for identifying languages across diverse data modalities. The model architecture employs logistic regression to enhance interpretability and feature extraction from each input modality. Trained on a comprehensive multilingual dataset, the model exhibits robust performance, showcasing its applicability to real-world scenarios. The model’s ability to process text, files, and images makes it well-suited for applications in content filtering, cross-modal information retrieval, and multilingual sentiment analysis. This research contributes to the advancement of language detection models by offering a unified solution for handling diverse input types.

Annotate and retrieve in vivo images using hybrid self-organizing map

Multimodal retrieval has gained much attention lately due to its effectiveness over uni-modal retrieval. For instance, visual features often under-constrain the description of an image in content-based retrieval; however, another modality, such as collateral text, can be introduced to abridge the semantic gap and make the retrieval process more efficient. This article proposes the application of cross-modal fusion and retrieval on real in vivo gastrointestinal images and linguistic cues, as the visual features alone are insufficient for image description and to assist gastroenterologists. So, a cross-modal information retrieval approach has been proposed to retrieve related images given text and vice versa while handling the heterogeneity gap issue among the modalities. The technique comprises two stages: (1) individual modality feature learning; and (2) fusion of two trained networks. In the first stage, two self-organizing maps (SOMs) are trained separately using images and texts, which are clustered in the respective SOMs based on their similarity. In the second (fusion) stage, the trained SOMs are integrated using an associative network to enable cross-modal retrieval. The underlying learning techniques of the associative network include Hebbian learning and Oja learning (Improved Hebbian learning). The introduced framework can annotate images with keywords and illustrate keywords with images, and it can also be extended to incorporate more diverse modalities. Extensive experimentation has been performed on real gastrointestinal images obtained from a known gastroenterologist that have collateral keywords with each image. The obtained results proved the efficacy of the algorithm and its significance in aiding gastroenterologists in quick and pertinent decision making.

<i>c</i>-SNE: Deep Cross-modal Retrieval based on Subjective Information using Stochastic Neighbor Embedding

Cross-modal information retrieval based on subjective information aims to enable flexible media retrieval services, such as allowing users to specify, for example, an image to search audio clips. The resulting audio clips should have an impression similar to the specified image. Existing methods focus on building cross-media cross-modal relationships using objective information (such as the standard caption). However, such a relation can be built only between the pieces of media that are originally related, which limits the flexibility of cross-modal media retrieval. This research leverages subjective information in the media clips for similarity calculation to achieve more flexibility. We propose a novel cross-modal stochastic neighbor embedding technique called c-SNE. c-SNE can extract features of subjective information from pieces of media and map them in the common embedding space. It is a learning technique to bridge the heterogeneous gap between the modal distributions using label-weighted SNE. It allows users to find the media that share the same subjective information with a query medium. Our experimental results on the benchmark datasets demonstrate that the proposed method effectively performs in cross-modal distribution alignment and retrieval. Furthermore, our user study with ten users with 600 data points confirmed that c-SNE outperforms three related methods in the actual usage situation from the users' perspective.

Improving visual-semantic embeddings by learning semantically-enhanced hard negatives for cross-modal information retrieval

Visual Semantic Embedding (VSE) networks aim to extract the semantics of images and their descriptions and embed them into the same latent space for cross-modal information retrieval. Most existing VSE networks are trained by adopting a hard negatives loss function which learns an objective margin between the similarity of relevant and irrelevant image–description embedding pairs. However, the objective margin in the hard negatives loss function is set as a fixed hyperparameter that ignores the semantic differences of the irrelevant image–description pairs. To address the challenge of measuring the optimal similarities between image–description pairs before obtaining the trained VSE networks, this paper presents a novel approach that comprises two main parts: (1) finds the underlying semantics of image descriptions; and (2) proposes a novel semantically-enhanced hard negatives loss function, where the learning objective is dynamically determined based on the optimal similarity scores between irrelevant image–description pairs. Extensive experiments were carried out by integrating the proposed methods into five state-of-the-art VSE networks that were applied to three benchmark datasets for cross-modal information retrieval tasks. The results revealed that the proposed methods achieved the best performance and can also be adopted by existing and future VSE networks.

Analysis of Network Information Retrieval Method Based on Metadata Ontology

In order to solve the problem that people can accurately search for the network information they need, the research on network information retrieval methods becomes more important. This article is mainly about the research of network information retrieval methods based on metadata ontology calculations. This article constructs an LDA three-layer Bayesian model with a three-layer structure of document, topic, and single order. The three-layer structure obeys a random polynomial distribution and can be calculated the joint distribution probability of all variables in the LDA model greatly increases the calculation efficiency. Using a cross-modal information retrieval method, it can mine the common data features between different modal data and analyze the semantic correlation between different modal data, improve the accuracy of search, and solve the existence of different modal data. There is a gap in the expression semantics between heterogeneous and different modal data. The experimental results in this paper show that the text feature extraction of the network information retrieval method based on the metadata ontology calculation has a good performance in terms of accuracy, and the accuracy of the extraction and clustering results is as high as about 90%. The improved CCA algorithm used is better than the traditional CCA and the accuracy is improved by 23%, which is 12% higher than the LDA-CCA algorithm.

Graph Convolutional Networks for Cross-Modal Information Retrieval

In recent years, due to the wide application of deep learning and more modal research, the corresponding image retrieval system has gradually extended from traditional text retrieval to visual retrieval combined with images and has become the field of computer vision and natural language understanding and one of the important cross-research hotspots. This paper focuses on the research of graph convolutional networks for cross-modal information retrieval and has a general understanding of cross-modal information retrieval and the related theories of convolutional networks on the basis of literature data. Modal information retrieval is designed to combine high-level semantics with low-level visual capabilities in cross-modal information retrieval to improve the accuracy of information retrieval and then use experiments to verify the designed network model, and the result is that the model designed in this paper is more accurate than the traditional retrieval model, which is up to 90%.

Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders

Despite the evolution of deep-learning-based visual-textual processing systems, precise multi-modal matching remains a challenging task. In this work, we tackle the task of cross-modal retrieval through image-sentence matching based on word-region alignments, using supervision only at the global image-sentence level. Specifically, we present a novel approach called Transformer Encoder Reasoning and Alignment Network (TERAN). TERAN enforces a fine-grained match between the underlying components of images and sentences (i.e., image regions and words, respectively) to preserve the informative richness of both modalities. TERAN obtains state-of-the-art results on the image retrieval task on both MS-COCO and Flickr30k datasets. Moreover, on MS-COCO, it also outperforms current approaches on the sentence retrieval task. Focusing on scalable cross-modal information retrieval, TERAN is designed to keep the visual and textual data pipelines well separated. Cross-attention links invalidate any chance to separately extract visual and textual features needed for the online search and the offline indexing steps in large-scale retrieval systems. In this respect, TERAN merges the information from the two domains only during the final alignment phase, immediately before the loss computation. We argue that the fine-grained alignments produced by TERAN pave the way toward the research for effective and efficient methods for large-scale cross-modal information retrieval. We compare the effectiveness of our approach against relevant state-of-the-art methods. On the MS-COCO 1K test set, we obtain an improvement of 5.7% and 3.5% respectively on the image and the sentence retrieval tasks on the Recall@1 metric. The code used for the experiments is publicly available on GitHub at https://github.com/mesnico/TERAN .

On the Limitations of Visual-Semantic Embedding Networks for Image-to-Text Information Retrieval.

Visual-semantic embedding (VSE) networks create joint image–text representations to map images and texts in a shared embedding space to enable various information retrieval-related tasks, such as image–text retrieval, image captioning, and visual question answering. The most recent state-of-the-art VSE-based networks are: VSE++, SCAN, VSRN, and UNITER. This study evaluates the performance of those VSE networks for the task of image-to-text retrieval and identifies and analyses their strengths and limitations to guide future research on the topic. The experimental results on Flickr30K revealed that the pre-trained network, UNITER, achieved 61.5% on average Recall@5 for the task of retrieving all relevant descriptions. The traditional networks, VSRN, SCAN, and VSE++, achieved 50.3%, 47.1%, and 29.4% on average Recall@5, respectively, for the same task. An additional analysis was performed on image–text pairs from the top 25 worst-performing classes using a subset of the Flickr30K-based dataset to identify the limitations of the performance of the best-performing models, VSRN and UNITER. These limitations are discussed from the perspective of image scenes, image objects, image semantics, and basic functions of neural networks. This paper discusses the strengths and limitations of VSE networks to guide further research into the topic of using VSE networks for cross-modal information retrieval tasks.

Clustering-driven Deep Adversarial Hashing for scalable unsupervised cross-modal retrieval

With the advent of the big data era, multimedia data is growing rapidly, and its data modalities is also becoming diversified. Therefore, the demand for the speed and accuracy of cross-modal information retrieval is increasing. Hashing-based cross-modal retrieval technology attracts widespread attention, it encodes multimedia data into a common binary hash space, thereby effectively measuring the correlation between samples from different modalities. In this paper, we propose a novel end-to-end deep cross-modal retrieval framework, namely Clustering-driven Deep Adversarial Hashing (CDAH), which has three main characteristics. Firstly, CDAH learns discriminative clusters recursively through a soft clustering model. It attempts to generate modal-invariant representations in a common space by obfuscating the modality classifier, which tries to distinguish different modalities according to the generated representations. Secondly, in order to minimize the modal gap between feature representations from different modalities with the same semantic label, and to maximize the distance between images and texts with different labels, CDAH constructs a fused-semantics matrix to integrate the original domain information from different modalities, serving as self-supervised information to refine the binary codes. Finally, CDAH skillfully uses a scaled tanh function to adaptively learn the binary codes, which will gradually converge to the original tricky binary coding problem. We conduct comprehensive experiments on four popular datasets, and the experimental results demonstrate the superiority of our model against the state-of-the-art methods.

Functional heterogeneity in the left lateral posterior parietal cortex during visual and haptic crossmodal dot-surface matching.

BackgroundVision and touch are thought to contribute information to object perception in an independent but complementary manner. The left lateral posterior parietal cortex (LPPC) has long been associated with multisensory information processing, and it plays an important role in visual and haptic crossmodal information retrieval. However, it remains unclear how LPPC subregions are involved in visuo‐haptic crossmodal retrieval processing.MethodsIn the present study, we used an fMRI experiment with a crossmodal delayed match‐to‐sample paradigm to reveal the functional role of LPPC subregions related to unimodal and crossmodal dot‐surface retrieval.ResultsThe visual‐to‐haptic condition enhanced the activity of the left inferior parietal lobule relative to the haptic unimodal condition, whereas the inverse condition enhanced the activity of the left superior parietal lobule. By contrast, activation of the left intraparietal sulcus did not differ significantly between the crossmodal and unimodal conditions. Seed‐based resting connectivity analysis revealed that these three left LPPC subregions engaged distinct networks, confirming their different functions in crossmodal retrieval processing.ConclusionTaken together, the findings suggest that functional heterogeneity of the left LPPC during visuo‐haptic crossmodal dot‐surface retrieval processing reflects that the left LPPC does not simply contribute to retrieval of past information; rather, each subregion has a specific functional role in resolving different task requirements.

Cross-modal learning with prior visual relation knowledge

Visual relational reasoning is a central component in recent cross-modal analysis tasks, which aims at reasoning about the visual relationships between objects and their properties. These relationships provide rich semantics and help to enhance the visual representation for improving cross-modal learning. Previous works have succeeded in modeling latent visual relationships or rigid-categorized visual relationships. However, these kinds of methods leave out the problem of ambiguity inherent in the visual relationships because of the diverse relational semantics of different visual appearances. In this work, we explore to model the visual relationships by context-aware representations based on human prior knowledge. Based on such representations, we novelly propose a plug-and-play visual relational reasoning module to enhance image encoding. Specifically, we design an Anisotropic Graph Convolution to utilize the information of relation embeddings and relation directionality between objects for generating relation-aware image representations. We demonstrate the effectiveness of the relational reasoning module by applying it to both Visual Question Answering (VQA) and Cross-Modal Information Retrieval (CMIR) tasks. Extensive experiments are conducted on VQA 2.0 and CMPlaces datasets and superior performance is reported when comparing with state-of-the-art works.

Patient trial matching using pseudo-siamese network.

e14050 Background: Clinical trials suffer from insufficient patient (pt) recruitment. The availability of electronic health records (EHR) and trial eligibility criteria (EC) is promising for data driven pt-trial matching. The objective is to find qualified pt given patients' EHR and trial EC in unstructured text EC. Pseudo-Siamese network is a novel subfield within information retrieval and has shown great success in the cross-modal information retrieval problems such as semantic image-text retrieval (e.g., match images with text descriptions). The objective is to find the match between pts and clinical trials using Pseudo-Siamese network based cross-modal retrieval. Our model addresses the following challenges: (1) How to match unstructured EC text with structured EHR where EC often encode more general disease concepts and EHR represent pt conditions using more specific medical codes. (2) How to capture pts' evolving health conditions. (3) How to explicitly handle the difference for inclusion and exclusion criteria. Methods: Our matching model addresses these challenges as follows: (1) we augment the medical codes in pts’ records with their textual descriptions and hierarchical taxonomies, such that concepts can be embedded in finer and more coarse levels for better concept alignment across pt data and ECs. (2) We include an attentive dynamic memory network that extracts the best matching and more recent pt EHR to match with ECs. (3) We introduce a composite loss term to maximize the similarity between pt records and inclusion criteria while minimizes the similarity between pt records and exclusion criteria. Results: We evaluated our model on a pt-trial match dataset on the ECs collected from 590 clinical trials from ClinicalTrials.gov. We also extract 83,371 pt claims data from IQVIA database collected (2002-2018), where each pt is eligible for at least one trial. We compared our model with leading pt-trial matching models. Our model significantly outperforms the best baseline model by 24.3% relatively higher accuracy score. We also tested these models in 34 oncology trials in 25 cancers. Results will be reported. Conclusions: Pseudo-Siamese network successfully solved the cross-modal information retrieval problems. We therefore propose a new pt-trial matching model based on Pseudo-Siamese network model. Experiments on real-world datasets demonstrated that our model significantly outperforms existing works in pt-trial matching for oncology trials.

Reasoning on the Relation: Enhancing Visual Representation for Visual Question Answering and Cross-Modal Retrieval

Cross-modal analysis has become a promising direction for artificial intelligence. Visual representation is crucial for various cross-modal analysis tasks that require visual content understanding. Visual features which contain semantical information can disentangle the underlying correlation between different modalities, thus benefiting the downstream tasks. In this paper, we propose a Visual Reasoning and Attention Network (VRANet) as a plug-and-play module to capture rich visual semantics and help to enhance the visual representation for improving cross-modal analysis. Our proposed VRANet is built based on the bilinear visual attention module which identifies the critical objects. We propose a novel Visual Relational Reasoning (VRR) module to reason about pair-wise and inner-group visual relationships among objects guided by the textual information. The two modules enhance the visual features at both relation level and object level. We demonstrate the effectiveness of the proposed VRANet by applying it to both Visual Question Answering (VQA) and Cross-Modal Information Retrieval (CMIR) tasks. Extensive experiments conducted on VQA 2.0, CLEVR, CMPlaces, and MS-COCO datasets indicate superior performance comparing with state-of-the-art work.

CMIR-NET : A deep learning based model for cross-modal retrieval in remote sensing

We address the problem of cross-modal information retrieval in the domain of remote sensing. In particular, we are interested in two application scenarios: i) cross-modal retrieval between panchromatic (PAN) and multispectral imagery, and ii) multi-label image retrieval between very high resolution (VHR) images and speech-based label annotations. These multi-modal retrieval scenarios are more challenging than the traditional uni-modal retrieval approaches given the inherent differences in distributions between the modalities. However, with the increasing availability of multi-source remote sensing data and the scarcity of enough semantic annotations, the task of multi-modal retrieval has recently become extremely important. In this regard, we propose a novel deep neural network-based architecture that is considered to learn a discriminative shared feature space for all the input modalities, suitable for semantically coherent information retrieval. Extensive experiments are carried out on the benchmark large-scale PAN - multispectral DSRSID dataset and the multi-label UC-Merced dataset. Together with the Merced dataset, we generate a corpus of speech signals corresponding to the labels. Superior performance with respect to the current state-of-the-art is observed in all the cases.

Learning cross-modal correlations by exploring inter-word semantics and stacked co-attention

Cross-modal information retrieval aims to find heterogeneous data of various modalities from a given query of one modality. The main challenge is to learn the semantic correlations between different modalities and measure the distance across modalities. For text-image retrieval, existing work mostly uses off-the-shelf Convolutional Neural Network (CNN) for image feature extraction. For texts, word-level features such as bag-of-words or word2vec are employed to build deep learning models to represent texts. Besides word-level semantics, the semantic relations between words are also informative but less explored. In this paper, we explore the inter-word semantics by modelling texts by graphs using similarity measure based on word2vec. Besides feature presentations, we further study the problem of information imbalance between different modalities when describing the same semantics. For example textual descriptions often contain more background information that cannot be conveyed by images and vice versa. We propose a stacked co-attention network to progressively learn the mutually attended features of different modalities and enhance their fine-grained correlations. A dual-path neural network is proposed for cross-modal information retrieval. The model is trained by a pairwise similarity loss function to maximize the similarity of relevant text-image pairs and minimize the similarity of irrelevant pairs. Experimental results show that the proposed model outperforms the state-of-the-art methods significantly, with 19% improvement on accuracy for the best case.

Parallel text alignment

Parallel Text Alignment (PTA) is the automatic content alignment of multiple text documents originating or derived from the same source. Such an alignment allows for more powerful retrieval of document contents due to the addition of translation approaches to the retrieval problem. The implications of this result in improving multimedia data access in digital library applications, range from facilitating the analysis of multiple English language translations of classical texts to enabling on-demand and random comparison of multiple transcriptions derived from a given audio stream or associated with a given stream of video, audio, or images. In this paper we present an efficient algorithm for achieving such an alignment, and demonstrate its use with two applications. This result is an application of the new framework of Cross-Modal Information Retrieval recently developed at Dartmouth College and Michigan State University.