Multimodal Knowledge Research Articles

Scene-Driven Multimodal Knowledge Graph Construction for Embodied AI

Scene-Driven Multimodal Knowledge Graph Construction for Embodied AI

Text-and-Image Learning Transformer for Cross-modal Person Re-identification

Text-based person re-identification aims to find the target person from a large pedestrian gallery with the given natural language description. Previous works mainly focus on embedding salient textual and visual representations in a common latent space by utilizing the dual-path structure or parameter-shared network. However, they still lack the ability to effectively extract fine-grained unimodal features as well as fuse the cross-modal data, leading to the increase of misaligned cases. To settle these issues, we propose a text-and-image implicit learning Transformer (TILT) to eliminate textual anisotropy and enhance the cross-modal alignment from both domains based on the bi-direction multimodal encoders. Specifically, we apply the pre-trained multimodal embedding module to overcome the unimodal anisotropy problem with contrastive learning, and map fine-grained features with dual encoder in bidirectional masking. Then, we design the cross-modal interaction encoder to comprehensively mine implicit cross-modal relations by reconstructing masked tokens, and fuse rich multi-modal knowledge in a common space. In addition, the cross-modal similarity matching module is proposed to optimize the intra-domain classification and decrease the inter-domain divergence. Extensive experiments are conducted on three public benchmarks CUHK-PEDES, ICFG-PEDES and RSTPReid to verify the effectiveness of our proposed framework. And results prove that our model outperforms state-of-the-art methods on all metrics.

Enhancing ophthalmology medical record management with multi-modal knowledge graphs

The electronic medical record management system plays a crucial role in clinical practice, optimizing the recording and management of healthcare data. To enhance the functionality of the medical record management system, this paper develops a customized schema designed for ophthalmic diseases. A multi-modal knowledge graph is constructed, which is built upon expert-reviewed and de-identified real-world ophthalmology medical data. Based on this data, we propose an auxiliary diagnostic model based on a contrastive graph attention network (CGAT-ADM), which uses the patient’s diagnostic results as anchor points and achieves auxiliary medical record diagnosis services through graph clustering. By implementing contrastive methods and feature fusion of node types, text, and numerical information in medical records, the CGAT-ADM model achieved an average precision of 0.8563 for the top 20 similar case retrievals, indicating high performance in identifying analogous diagnoses. Our research findings suggest that medical record management systems underpinned by multimodal knowledge graphs significantly enhance the development of AI services. These systems offer a range of benefits, from facilitating assisted diagnosis and addressing similar patient inquiries to delving into potential case connections and disease patterns. This comprehensive approach empowers healthcare professionals to garner deeper insights and make well-informed decisions.

ChemAP: predicting drug approval with chemical structures before clinical trial phase by leveraging multi-modal embedding space and knowledge distillation

Recent studies showed that the likelihood of drug approval can be predicted with clinical data and structure information of drug using computational approaches. Predicting the likelihood of drug approval can be innovative and of high impact. However, models that leverage clinical data are applicable only in clinical stages, which is not very practical. Prioritizing drug candidates and early-stage decision-making in the de novo drug development process is crucial in pharmaceutical research to optimize resource allocation. For early-stage decision-making, we need a computational model that uses only chemical structures. This seemingly impossible task may utilize the predictive power with multi-modal features including clinical data. In this work, we introduce ChemAP (Chemical structure-based drug Approval Predictor), a novel deep learning scheme for drug approval prediction in the early-stage drug discovery phase. ChemAP aims to enhance the possibility of early-stage decision-making by enriching semantic knowledge to fill in the gap between multi-modal and single-modal chemical spaces through knowledge distillation techniques. This approach facilitates the effective construction of chemical space solely from chemical structure data, guided by multi-modal knowledge related to efficacy, such as clinical trials and patents of drugs. In this study, ChemAP achieved state-of-the-art performance, outperforming both traditional machine learning and deep learning models in drug approval prediction, with AUROC and AUPRC scores of 0.782 and 0.842 respectively on the drug approval benchmark dataset. Additionally, we demonstrated its generalizability by outperforming baseline models on a recent external dataset, which included drugs from the 2023 FDA-approved list and the 2024 clinical trial failure drug list, achieving AUROC and AUPRC scores of 0.694 and 0.851. These results demonstrate that ChemAP is an effective method in predicting drug approval only with chemical structure information of drug so that decision-making can be done at the early stages of drug development process. To the best of our knowledge, our work is the first of its kind to show that prediction of drug approval is possible only with structure information of drug by defining the chemical space of approved and unapproved drugs using deep learning technology.

SDDA: A progressive self-distillation with decoupled alignment for multimodal image-text classification

Multimodal image-text classification endeavors to deduce the correct category based on the information encapsulated in image-text pairs. Despite the commendable performance achieved by current image-text methodologies, the intrinsic multimodal heterogeneity persists as a challenge, with the contributions from diverse modalities exhibiting considerable variance. In this study, we address this issue by introducing a novel decoupled multimodal Self-Distillation (SDDA) approach, aimed at facilitating fine-grained alignment of shared and private features of image-text features in a low-dimensional space, thereby reducing information redundancy. Specifically, each modality representation is decoupled in an autoregressive manner into two segments within a modality-irrelevant/exclusive space. SDDA imparts additional knowledge transfer to each decoupled segment via self-distillation, while also offering flexible, richer multimodal knowledge supervision for unimodal features. Multimodal classification experiments conducted on two publicly available benchmark datasets verified the efficacy of the algorithm, demonstrating that SDDA surpasses the state-of-the-art baselines.

Overcoming language priors in visual question answering with cumulative learning strategy

The performance of visual question answering(VQA) has witnessed great progress over the last few years. However, many current VQA models tend to rely on superficial linguistic correlations between questions and answers, often failing to sufficiently learn multi-modal knowledge from both vision and language, and thus suffering significant performance drops. To address this issue, the VQA-CP v2.0 dataset was developed to reduce language biases by greedily re-partitioning the distribution of VQA v2.0’s training and test sets. According to the fact that achieving high performance on real-world datasets requires effective learning from minor classes, in this paper we analyze the presence of skewed long-tail distributions in the VQA-CP v2.0 dataset and propose a new ensemble-based parameter-insensitive framework. This framework is built on two representation learning branches and a joint learning block, which are designed to reduce language biases in VQA tasks. Specifically, the representation learning branches can ensure the superior representative ability learned from the major and minor classes. The joint learning block forces the model to initially concentrate on major classes for robust representation and then gradually shifts its focus towards minor classes for classification during the training progress. Experimental results demonstrate that our approach outperforms the state-of-the-art works on the VQA-CP v2.0 dataset without requiring additional annotations. Notably, on the “num” type, our framework exceeds the second-best method (without extra annotations) by 8.64%. Meanwhile, our approach does not sacrifice accuracy performance on the VQA v2.0 dataset compared with the baseline model.

From text to mask: Localizing entities using the attention of text-to-image diffusion models

Diffusion models have revolted the field of text-to-image generation recently. The unique way of fusing text and image information contributes to their remarkable capability of generating highly text-related images. From another perspective, these generative models imply clues about the precise correlation between words and pixels. This work proposes a simple but effective method to utilize the attention mechanism in the denoising network of text-to-image diffusion models. Without additional training time nor inference-time optimization, the semantic grounding of phrases can be attained directly. We evaluate our method on Pascal VOC 2012 and Microsoft COCO 2014 under weakly-supervised semantic segmentation setting and our method achieves superior performance to prior methods. In addition, the acquired word-pixel correlation is generalizable for the learned text embedding of customized generation methods, requiring only a few modifications. To validate our discovery, we introduce a new practical task called “personalized referring image segmentation” with a new dataset. Experiments in various situations demonstrate the advantages of our method compared to strong baselines on this task. In summary, our work reveals a novel way to extract the rich multi-modal knowledge hidden in diffusion models for segmentation.

A radial basis deformable residual convolutional neural model embedded with local multi-modal feature knowledge and its application in cross-subject classification

In the spatial cognition and emotion recognition tasks based on electroencephalography (EEG), the signal information representation of the single modal is incomplete because of the significant inter-subject differences in the EEG signals, resulting in low generalization performance of classification models. To address this issue, we propose a radial basis deformable residual convolutional neural networks model embedded with local multi-modal feature knowledge (RBDRCNN-LMFK). The RBDRCNN leverages Euclidean distance alignment, deformable convolution, depth-separable convolution, and residual connection modules for effective EEG feature extraction. The embedded local feature knowledge algorithm enables the effective combination of multi-modal information. To further validate the effectiveness of the algorithm, we used the left-one-subject-out cross-validation algorithm on the virtual city walking (VCW), SJTU Emotion EEG Dataset IV (SEED IV), and Building block gesture recognition (BBGR) datasets for spatial cognition and emotion recognition tasks. The average accuracy of the VCW was 97.84 % using the kNN classifier, surpassing the Concate fusion method’s 96.62 %. The average accuracy for the SEED IV dataset was 87.56 %, higher than the Concate Fusion’s 69.97 %. On the BBGR dataset, the kNN classifier achieved an average accuracy of 87.80 %, compared to 85.31 % with the Concate fusion. The results show that the model enhances the recognition of biologically significant features in EEG signals by embedding local features and increases the correlation between different brain regions associated with the task. This work illustrates a promising direction of using deep learning models to discover effective task-related features from highly diverse EEG signals and enhance brain regions’ correlation through multi-modal knowledge embedding.

ICKA: An instruction construction and Knowledge Alignment framework for Multimodal Named Entity Recognition

Multimodal Named Entity Recognition (MNER) aims to identify entities of predefined types in text by leveraging information from multiple modalities, most notably textual and visual information. Most efforts concentrate on improving cross-modality attention mechanisms to facilitate guidance between modalities. However, they still suffer from certain limitations: (1) it is difficult to establish a unified representation to bridge the semantic gap among different modalities; (2) mining the implicit relationships between text and image is crucial yet challenging. In this paper, we propose an Instruction Construction and Knowledge Alignment Framework for MNER named ICKA to address these issues. Specifically, we first employ a multi-head cross-modal attention mechanism to obtain the cross-modal fusion representation by fusing features from text–image pairs. Then, we integrate external knowledge from the pre-trained vision-language model (VLM) to facilitate semantic alignment between text and image and obtain inter-modality connections. Next, we construct the multimodal instruction that consists of the modal features and uses the inter-modality connections as a bridge between them. We then integrate the instruction into the language model to effectively incorporate multimodal knowledge. Finally, we perform sequence labeling using a Conditional Random Fields (CRF) decoder with a gating mechanism. The proposed method achieves F1 scores of 75.42% on the Twitter2015 dataset and 87.12% on the Twitter2017 dataset, demonstrating the competitiveness of our method.

Unpaired Image-text Matching via Multimodal Aligned Conceptual Knowledge.

Recently, the accuracy of image-text matching has been greatly improved by multimodal pretrained models, all of which use millions or billions of paired images and texts for supervised model learning. Different from them, human brains can well match images with texts using their stored multimodal knowledge. Inspired by that, this paper studies a new scenario as unpaired image-text matching, in which paired images and texts are assumed to be unavailable during model learning. To deal with it, we accordingly propose a simple yet effective method namely Multimodal Aligned Conceptual Knowledge (MACK). First, we collect a set of words and their related image regions from publicly available datasets, and compute prototypical region representations to obtain pretrained general knowledge. To make the obtained knowledge better suit for certain datasets, we refine it using unpaired images and texts in a self-supervised learning manner to obtain fine-tuned domain knowledge. Then, to match given images with texts based on the knowledge, we represent parsed words in the texts by prototypical region representations, and compute region-word similarity scores. At last, the scores are aggregated based on bidirectional similarity pooling into an image-text similarity score, which can be directly used for unpaired image-text matching. The proposed MACK is complementary with existing models, which can be easily extended as a re-ranking method to substantially improve their performance of zero-shot and cross-dataset image-text matching.

Triplet-aware graph neural networks for factorized multi-modal knowledge graph entity alignment

Multi-Modal Entity Alignment (MMEA), aiming to discover matching entity pairs on two multi-modal knowledge graphs (MMKGs), is an essential task in knowledge graph fusion. Through mining feature information of MMKGs, entities are aligned to tackle the issue that an MMKG is incapable of effective integration. The recent attempt at neighbors and attribute fusion mainly focuses on aggregating multi-modal attributes, neglecting the structure effect with multi-modal attributes for entity alignment. This paper proposes an innovative approach, namely TriFac, to exploit embedding refinement for factorizing the original multi-modal knowledge graphs through a two-stage MMKG factorization. Notably, we propose triplet-aware graph neural networks to aggregate multi-relational features. We propose multi-modal fusion for aggregating multiple features and design three novel metrics to measure knowledge graph factorization performance on the unified factorized latent space. Empirical results indicate the effectiveness of TriFac, surpassing previous state-of-the-art models on two MMEA datasets and a power system dataset.

Open-category referring expression comprehension via multi-modal knowledge transfer

Referring expression comprehension (REC) is a challenging task that involves locating a particular object in an image based on a natural language query. Despite REC showing potential for identifying objects beyond a fixed set of predefined categories, existing models display limited accuracy when confronted with categories not seen during training. To overcome this limitation, in this work, we introduce a new setting called Open-Category Referring Expression Comprehension that focuses more on model generalization capabilities on unseen categories, and present an Multi-modal Knowledge Transfer REC (MTKREC) framework to address this challenge. Specifically, to handle various novel categories, our framework initially constructs an isolated proposal embedding method that integrates pre-training knowledge from CLIP. This method isolates object proposals by cropping them, passing them to CLIP for box-level embedding, and concurrently obtaining box-level proposal embedding from Faster-RCNN. Then, inspired by ResNet, our framework proposes a Residual Self-Attention (RSA) strategy within the fusion module to maximize the utilization of information from the isolated proposal embedding method. To further bolster the model’s capabilities, we transfer knowledge from UNITER by reusing its parameters during the multi-modal fusion process, and explore knowledge distillation techniques to accelerate the model’s performance. We also construct new datasets sub-sampled from RefCOCO, RefCOCO+, and RefCOCOg datasets, that enable evaluation for our model. Extensive experiments on new datasets demonstrate the effectiveness of our framework.

Knowledge Management Model for Urban Flood Emergency Response Based on Multimodal Knowledge Graphs

Recently, frequent flood disasters in China have seriously threatened economic development and public safety. This paper addresses the need for a dynamic urban flood emergency knowledge management system in emergency management departments and the lack of systematic knowledge among emergency managers regarding urban flood control. A multimodal knowledge graph-based urban flood emergency knowledge management model was constructed to enhance the decision-making capabilities of emergency management departments, improve the efficiency of public emergency evacuation, and reduce losses from urban flood disasters by analyzing the shortcomings of the existing emergency management system. An intelligent and dynamic flood emergency knowledge management model was built. This paper integrates multimodal knowledge graph technology to establish a multimodal emergency knowledge management framework for urban flood control. It develops and simulates the proposed model’s application scenarios for urban flood emergency evacuation using the Flocking algorithm on the NetLogo platform. Through simulation experiments, the practicality and effectiveness of the model in real flood disaster situations were examined, particularly in simulating crowd evacuation behavior. The study found that the model significantly improves the accuracy of information and decision-making speed during emergency responses and supports emergency management departments in conducting targeted and personalized emergency decisions. This research provides a scientific basis for emergency management departments to optimize their emergency response strategies to flood disasters and serves as a reference and example for the application of multimodal knowledge graph technology in emergency management.

M3KGR: A momentum contrastive multi-modal knowledge graph learning framework for recommendation

In recent years, there has been a discernible upswing in the utilization of knowledge graphs within recommender systems. This heightened interest in knowledge graphs stems from their ability to vividly illustrate relationships between items through augmented attribute representations. Scholars have endeavored to incorporate multi-modal data, encompassing texts and images, into knowledge graphs, albeit with restricted success. These efforts primarily hinge on image models for extracting visual representations and text models for extracting textual representations, prompting apprehensions regarding potential disparities between the two modalities. Moreover, the current approaches for computing node similarities using multi-modal information have not witnessed noteworthy progress or enhancements.Momentum Contrastive Multi-Modal Knowledge Graph Learning Framework for Recommendation (M3KGR) proposed in this paper aims to enhance the utilization of multi-modal data. Specifically, we leverage the existing CLIP model to simultaneously extract textual and visual representations of an entity, thus overcoming the challenge of integrating multi-modal information. Additionally, we introduce a multi-modal enhanced attention technique to further enhance the performance of the graph attention network. Furthermore, we introduce Momentum Contrast to mitigate the common issues of data sparsity and noise in recommender systems. Our comprehensive experiments conducted on three real datasets demonstrate that our model enhances the efficacy of multi-modal knowledge graphs in recommender systems by fully leveraging the potential of multi-modal data. The implementation of our model will be available at https://github.com/KlaineWei/M3KGR.

Multimodal Co-attention Fusion Network with Online Data Augmentation for Cancer Subtype Classification.

It is an essential task to accurately diagnose cancer subtypes in computational pathology for personalized cancer treatment. Recent studies have indicated that the combination of multimodal data, such as whole slide images (WSIs) and multi-omics data, could achieve more accurate diagnosis. However, robust cancer diagnosis remains challenging due to the heterogeneity among multimodal data, as well as the performance degradation caused by insufficient multimodal patient data. In this work, we propose a novel multimodal co-attention fusion network (MCFN) with online data augmentation (ODA) for cancer subtype classification. Specifically, a multimodal mutual-guided co-attention (MMC) module is proposed to effectively perform dense multimodal interactions. It enables multimodal data to mutually guide and calibrate each other during the integration process to alleviate inter- and intra-modal heterogeneities. Subsequently, a self-normalizing network (SNN)-Mixer is developed to allow information communication among different omics data and alleviate the high-dimensional small-sample size problem in multi-omics data. Most importantly, to compensate for insufficient multimodal samples for model training, we propose an ODA module in MCFN. The ODA module leverages the multimodal knowledge to guide the data augmentations of WSIs and maximize the data diversity during model training. Extensive experiments are conducted on the public TCGA dataset. The experimental results demonstrate that the proposed MCFN outperforms all the compared algorithms, suggesting its effectiveness.

WuMKG: a Chinese painting and calligraphy multimodal knowledge graph

Chinese Painting and Calligraphy (ChP&C) holds significant cultural value, representing integral aspects of both Chinese culture and global art. A considerable amount of ChP&C works are dispersed worldwide. With the emergence of digital humanities, a vast collection of cultural artifacts data is now available online. However, the online databases of these artifacts remain decentralized and diverse, posing significant challenges to their effective organization and utilization. Addressing this, our paper focuses on the Wu Men School of Painting and proposes a framework for constructing a multimodal knowledge graph for the ChP&C domain. We construct the domain ontology by analyzing the ChP&C knowledge schema. Then, we acquire knowledge from diverse data sources, including textual and visual information. To enhance data collection around collecting historical context and subject matter, we propose methods for seal extraction and subject extraction specific to ChP&C. We validate the effectiveness of these methods on the constructed dataset. Finally, we construct the Wu Men Multimodal Knowledge Graph (WuMKG) and implement applications such as cross-modal retrieval, knowledge-based question-answering and visualization.

TRAFMEL: Multimodal Entity Linking Based on Transformer Reranking and Multimodal Co-Attention Fusion

Multimodal entity linking aims to link mentions to target entities in the multimodal knowledge graph. The current multimodal entity linking mainly focuses on the global fusion of text and image, seldom fully exploring the correlation between modalities. In order to improve the fusion effect of multimodal feature, we propose a multimodal entity linking model based on a Multimodal Co-Attention Fusion strategy. This strategy is designed to enable text and image to guide each other for extracting features, thus making full exploration of the correlation between modalities to improve the fine-grained feature fusion effect. Furthermore, we also design a candidate entity generation strategy based on Transformer, which combines multiple candidate entity sets and adjusts the candidate entity ranking to obtain high-quality candidate entity sets. We perform experiments on domain datasets and public datasets, and the experimental results demonstrate that our model has a good performance in candidate entity generation and multimodal feature fusion, outperforming the state-of-the-art baseline models.

Knowledge is power: Open-world knowledge representation learning for knowledge-based visual reasoning

Knowledge-based visual reasoning requires the ability to associate outside knowledge that is not present in a given image for cross-modal visual understanding. Two deficiencies of the existing approaches are that (1) they only employ or construct elementary and explicit but superficial knowledge graphs while lacking complex and implicit but indispensable cross-modal knowledge for visual reasoning, and (2) they also cannot reason new/unseen images or questions in open environments and are often violated in real-world applications. How to represent and leverage tacit multimodal knowledge for open-world visual reasoning scenarios has been less studied. In this paper, we propose a novel open-world knowledge representation learning method to not only construct implicit knowledge representations from the given images and their questions but also enable knowledge transfer from a known given scene to an unknown scene for answer prediction. Extensive experiments conducted on six benchmarks demonstrate the superiority of our approach over other state-of-the-art methods. We apply our approach to other visual reasoning tasks, and the experimental results show that our approach, with its good performance, can support related reasoning applications.

HKA: A Hierarchical Knowledge Alignment Framework for Multimodal Knowledge Graph Completion

Recent years have witnessed the successful application of knowledge graph techniques in structured data processing, while how to incorporate knowledge from visual and textual modalities into knowledge graphs has been given less attention. To better organize them, Multimodal Knowledge Graphs (MKGs), comprising the structural triplets of traditional Knowledge Graphs (KGs) together with entity-related multimodal data (e.g., images and texts), have been introduced consecutively. However, it is still a great challenge to explore MKGs due to their inherent incompleteness. Although most existing Multimodal Knowledge Graph Completion (MKGC) approaches can infer missing triplets based on available factual triplets and multimodal information, they almost ignore the modal conflicts and supervisory effect, failing to achieve a more comprehensive understanding of entities. To address these issues, we propose a novel H ierarchical K nowledge A lignment ( HKA ) framework for MKGC. Specifically, a macro-knowledge alignment module is proposed to capture global semantic relevance between modalities for dealing with modal conflicts in MKG. Furthermore, a micro-knowledge alignment module is also developed to reveal the local consistency information through inter- and intra-modality supervisory effect more effectively. By integrating different modal predictions, a final decision can be made. Experimental results on three benchmark MKGC tasks have demonstrated the effectiveness of the proposed HKA framework.

MHD-Net: Memory-Aware Hetero-Modal Distillation Network for Thymic Epithelial Tumor Typing With Missing Pathology Modality.

Fusing multi-modal radiology and pathology data with complementary information can improve the accuracy of tumor typing. However, collecting pathology data is difficult since it is high-cost and sometimes only obtainable after the surgery, which limits the application of multi-modal methods in diagnosis. To address this problem, we propose comprehensively learning multi-modal radiology-pathology data in training, and only using uni-modal radiology data in testing. Concretely, a Memory-aware Hetero-modal Distillation Network (MHD-Net) is proposed, which can distill well-learned multi-modal knowledge with the assistance of memory from the teacher to the student. In the teacher, to tackle the challenge in hetero-modal feature fusion, we propose a novel spatial-differentiated hetero-modal fusion module (SHFM) that models spatial-specific tumor information correlations across modalities. As only radiology data is accessible to the student, we store pathology features in the proposed contrast-boosted typing memory module (CTMM) that achieves type-wise memory updating and stage-wise contrastive memory boosting to ensure the effectiveness and generalization of memory items. In the student, to improve the cross-modal distillation, we propose a multi-stage memory-aware distillation (MMD) scheme that reads memory-aware pathology features from CTMM to remedy missing modal-specific information. Furthermore, we construct a Radiology-Pathology Thymic Epithelial Tumor (RPTET) dataset containing paired CT and WSI images with annotations. Experiments on the RPTET and CPTAC-LUAD datasets demonstrate that MHD-Net significantly improves tumor typing and outperforms existing multi-modal methods on missing modality situations.