Multimodal Fusion Learning Research Articles

Progressive Pretraining Network for 3D System Matrix Calibration in Magnetic Particle Imaging.

Magnetic particle imaging (MPI) is an emerging technique for determining magnetic nanoparticle distributions in biological tissues. Although system-matrix (SM)-based image reconstruction offers higher image quality than the X-space-based approach, the SM calibration measurement is time-consuming. Additionally, the SM should be recalibrated if the tracer's characteristics or the magnetic field environment change, and repeated SM measurement further increase the required labor and time. Therefore, fast SM calibration is essential for MPI. Existing calibration methods commonly treat each row of the SM as independent of the others, but the rows are inherently related through the coil channel and frequency index. As these two elements can be regarded as additional multimodal information, we leverage the transformer architecture with a self-attention mechanism to encode them. Although the transformer has shown superiority in multimodal fusion learning across several fields, its high complexity may lead to overfitting when labeled data are scarce. Compared with labeled SM (i.e., full size), low-resolution SM data can be easily obtained, and fully using such data may alleviate overfitting. Accordingly, we propose a pseudo-label-based progressive pretraining strategy to leverage unlabeled data. Our method outperforms existing calibration methods on a public real-world OpenMPI dataset and simulation dataset. Moreover, our method improves the resolution of two in-house MPI scanners without requiring full-size SM measurements. Ablation studies confirm the contributions of modeling SM inter-row relations and the proposed pretraining strategy.

Multi-modal fusion learning through biosignal, audio, and visual content for detection of mental stress

Multi-modal fusion learning through biosignal, audio, and visual content for detection of mental stress

Prediction of mechanical properties of composite materials using multimodal fusion learning

Efficient and accurate measurement of material mechanical properties is important for material development. The mechanical properties of materials are comprehensively affected by factors such as material composition and microstructure. The existing single-mode prediction methods have the problems of poor mapping integrity and low prediction accuracy. In this paper, a multimodal fusion prediction model integrating material microstructure information and material composition information is proposed. Firstly, Convolutional neural network(CNN) is used to extract the material microstructure features, and Multilayer perceptron(MLP) is used to extract the component features. Then, the adaptive feature vector fusion module designed to adjust the influence of different modal information on the mechanical properties is used to achieve high-level feature fusion, and neural network is used to complete the mechanical properties prediction. The method is verified on the data of an alternative composite material. The method proposed provides a promising solution for the measurement and prediction of material properties.

MDTips: a multimodal-data-based drug-target interaction prediction system fusing knowledge, gene expression profile, and structural data.

Screening new drug-target interactions (DTIs) by traditional experimental methods is costly and time-consuming. Recent advances in knowledge graphs, chemical linear notations, and genomic data enable researchers to develop computational-based-DTI models, which play a pivotal role in drug repurposing and discovery. However, there still needs to develop a multimodal fusion DTI model that integrates available heterogeneous data into a unified framework. We developed MDTips, a multimodal-data-based DTI prediction system, by fusing the knowledge graphs, gene expression profiles, and structural information of drugs/targets. MDTips yielded accurate and robust performance on DTI predictions. We found that multimodal fusion learning can fully consider the importance of each modality and incorporate information from multiple aspects, thus improving model performance. Extensive experimental results demonstrate that deep learning-based encoders (i.e. Attentive FP and Transformer) outperform traditional chemical descriptors/fingerprints, and MDTips outperforms other state-of-the-art prediction models. MDTips is designed to predict the input drugs' candidate targets, side effects, and indications with all available modalities. Via MDTips, we reverse-screened candidate targets of 6766 drugs, which can be used for drug repurposing and discovery. https://github.com/XiaoqiongXia/MDTips and https://doi.org/10.5281/zenodo.7560544.

On Multi-modal Fusion Learning in constraint propagation

Constraint propagation methods demonstrate splendid performance in constrained clustering tasks. Although some multi-modal constraint propagation methods have been proposed in recent years, a feasible and robust approach to multi-modal feature fusion in pairwise constraint propagation is still in demand. This paper presents a novel multi-modal fusion approach in order to cope with the constraint propagation on multi-modal datasets, called Multi-modal Fusion Learning (MFL). The proposed method can reach a multi-modal fusion based on the observed constraint information and the propagation process. It is capable of handling any number of modalities without any prior knowledge of each modality. We merge the fusion learning and constraint propagation into one unified problem and solve it by a bound-constrained quadratic optimization. Our proposed method has been tested in clustering tasks on two publicly available multi-modal datasets to show its superior performance.