Multimodal Network Research Articles

Identifying chaotic dynamics in noisy time series through multimodal deep neural networks

Abstract Chaos detection is the problem of identifying whether a series of measurements is being sampled from an underlying set of chaotic dynamics. The unavoidable presence of measurement noise significantly affects the performance of chaos detectors, as discerning chaotic dynamics from stochastic signals becomes more challenging. This paper presents a computationally efficient multi-modal deep neural network tailored for chaos detection by combining information coming from the analysis of time series, recurrence plots and spectrograms. The proposed approach is the first one suitable for multi-class classification of chaotic systems while being robust with respect to measurement noise, and is validated on a dataset of 15 different chaotic and non-chaotic dynamics subject to white, pink or brown coloured noise.

Capturing impacts of travel preference on connected autonomous vehicle adoption of risk-averse travellers in multi-modal transportation networks

ABSTRACT This paper proposes a reliability-based combined modal split and traffic assignment model for the multi-modal transportation network with uncertainties in which the travellers' travel preferences, vehicle ownership, and the characteristics of connected autonomous vehicles (CAVs) are considered. Travellers are classified into captive travellers and choice travellers according to their preference for travel modes and vehicle ownership. The choice travellers need to make a tradeoff among the out-of-pocket cost, comfort, and travel time reliability and make decisions between continuing to commute by subway or purchasing a human-driven vehicle (HV) or CAV. The dogit-nested-logit model is adopted to characterise the effect of travel preference on the adoption of CAVs of the choice travellers, and the multinomial logit model is adopted to capture the effect of the differences between HV and CAV on information quality and value of travel time reliability on the path choice behaviour of travellers. The problem is formulated as an equivalent variational inequality model based on the properties of the equilibrium state, and its equivalency and solution existence are discussed. A path-based algorithm is employed for solving the problem, which combines the Monte Carlo simulation-based method and the column generation method. Numerical experiments are presented to illustrate the properties of the proposed models and validate the algorithm.

Multimodal fusion network for ICU patient outcome prediction

Over the past decades, massive Electronic Health Records (EHRs) have been accumulated in Intensive Care Unit (ICU) and many other healthcare scenarios. The rich and comprehensive information recorded presents an exceptional opportunity for patient outcome predictions. Nevertheless, due to the diversity of data modalities, EHRs exhibit a heterogeneous characteristic, raising a difficulty to organically leverage information from various modalities. It is an urgent need to capture the underlying correlations among different modalities. In this paper, we propose a novel framework named Multimodal Fusion Network (MFNet) for ICU patient outcome prediction. First, we incorporate multiple modality-specific encoders to learn different modality representations. Notably, a graph guided encoder is designed to capture underlying global relationships among medical codes, and a text encoder with pre-fine-tuning strategy is adopted to extract appropriate text representations. Second, we propose to pairwise merge multimodal representations with a tailored hierarchical fusion mechanism. The experiments conducted on the eICU-CRD dataset validate that MFNet achieves superior performance on mortality prediction and Length of Stay (LoS) prediction compared with various representative and state-of-the-art baselines. Moreover, comprehensive ablation study demonstrates the effectiveness of each component of MFNet.

Exploring the dynamic impact of the Northern Sea Route on China-Europe multimodal transportation competition in the context of the Belt and Road Initiative

As Arctic sea ice continues to melt, the Northern Sea Route (NSR) is anticipated to become the third major China-Europe transportation corridor, alongside the Suez Canal Route (SCR) and China Railway Express (CRE). This study investigates the integration of the NSR into the Belt and Road multimodal transportation network and analyzes its dynamic competition with traditional corridors across heterogeneous regions and markets in China. To achieve this, we developed a novel hybrid multi-criteria decision-making (MCDM) framework to evaluate the NSR's potential as an alternative China-Europe trade route. This framework introduces a new criteria utility function (CUF) and integrates the Bayesian best-worst method (BBWM) and measurement of alternatives and ranking according to compromise solution (MARCOS) method. This approach evaluates and prioritizes routes based on cost, time, reliability, safety, sustainability, and convenience, thereby detailing the NSR's market advantages and business prospects. The findings suggest that simply increasing the frequency of transportation services is insufficient to attract Chinese customers to the NSR. Instead, the route's competitiveness is primarily determined by transportation costs. In future scenarios, the NSR will offer additional route options for China-Europe containerized intermodal transport, potentially serving as a complementary corridor to the SCR and CRE under certain conditions. Coastal and sub-coastal regions north of Fujian Province are potential markets for the NSR, while inland regions far from seaports are better suited for connections via the CRE. Notably, time-insensitive containerized cargoes, such as industrial accessories, mechanical equipment, and metal crafts, exhibit higher potential demand for the NSR. In contrast, time-sensitive commodities like fashion apparel, electronics, and medical devices are more likely to be transported by cross-border railroads due to their urgency. Stakeholders, including government agencies, shipping companies, and exporters, can utilize these insights to optimize China-Europe transport organization and effectively commercialize the NSR.

Accessibility of methadone treatment via public transit for syringe services program participants in Miami-Dade County, Florida.

Methadone is an opioid receptor agonist medication used in the treatment of opioid use disorder (OUD). Geographic distance to opioid treatment programs (OTPs) is a major barrier to treatment, given requirements for direct observation of dosing and periodic drug screens, and 'methadone treatment deserts' are defined as a public transit threshold of 30 minutes. The purpose of this study was to examine public transit access to methadone treatment for participants of a syringe services program (SSP) in Miami-Dade County, Florida. Public transit times were calculated using the R library r5r, which facilitates multi-modal transportation network routing. General Transit Feed Specification data was combined with street network data from OpenStreetMap for Miami-Dade County. Transit times were estimated from the population-weighted centroid of each zip code (n=79) with participants of Miami's only SSP (n=1597) to the nearest OTP (n=4) using 10 departure windows aligned with OTP service hours. The mean one-way transit time from zip codes with SSP participants in Miami-Dade County to the nearest OTP was 80 minutes. 75 of the 79 (95%) zip codes with SSP participants in Miami-Dade County have a mean transit time to the closest OTP greater than 30 minutes. Transit times differ substantially between zip codes with different numbers of SSP participants, but not between departure windows. Nearly all zip codes with SSP participants in Miami-Dade County can be classified as 'methadone treatment deserts'. Geographic isolation of methadone treatment from public transit routes represents a significant barrier to equitable OUD treatment.

MGNDTI: A Drug-Target Interaction Prediction Framework Based on Multimodal Representation Learning and the Gating Mechanism.

Drug-Target Interaction (DTI) prediction facilitates acceleration of drug discovery and promotes drug repositioning. Most existing deep learning-based DTI prediction methods can better extract discriminative features for drugs and proteins, but they rarely consider multimodal features of drugs. Moreover, learning the interaction representations between drugs and targets needs further exploration. Here, we proposed a simple M ulti-modal G ating N etwork for DTI prediction, MGNDTI, based on multimodal representation learning and the gating mechanism. MGNDTI first learns the sequence representations of drugs and targets using different retentive networks. Next, it extracts molecular graph features of drugs through a graph convolutional network. Subsequently, it devises a multimodal gating network to obtain the joint representations of drugs and targets. Finally, it builds a fully connected network for computing the interaction probability. MGNDTI was benchmarked against seven state-of-the-art DTI prediction models (CPI-GNN, TransformerCPI, MolTrans, BACPI, CPGL, GIFDTI, and FOTF-CPI) using four data sets (i.e., Human, C. elegans, BioSNAP, and BindingDB) under four different experimental settings. Through evaluation with AUROC, AUPRC, accuracy, F1 score, and MCC, MGNDTI significantly outperformed the above seven methods. MGNDTI is a powerful tool for DTI prediction, showcasing its superior robustness and generalization ability on diverse data sets and different experimental settings. It is freely available at https://github.com/plhhnu/MGNDTI.

Deep learning of multimodal networks with topological regularization for drug repositioning

MotivationComputational techniques for drug-disease prediction are essential in enhancing drug discovery and repositioning. While many methods utilize multimodal networks from various biological databases, few integrate comprehensive multi-omics data, including transcriptomes, proteomes, and metabolomes. We introduce STRGNN, a novel graph deep learning approach that predicts drug-disease relationships using extensive multimodal networks comprising proteins, RNAs, metabolites, and compounds. We have constructed a detailed dataset incorporating multi-omics data and developed a learning algorithm with topological regularization. This algorithm selectively leverages informative modalities while filtering out redundancies.ResultsSTRGNN demonstrates superior accuracy compared to existing methods and has identified several novel drug effects, corroborating existing literature. STRGNN emerges as a powerful tool for drug prediction and discovery. The source code for STRGNN, along with the dataset for performance evaluation, is available at https://github.com/yuto-ohnuki/STRGNN.git.

The Multi-Objective Shortest Path Problem with Multimodal Transportation for Emergency Logistics

The optimization of emergency logistical transportation is crucial for the timely dispatch of aid and support to affected areas. By incorporating practical constraints into emergency logistics, this study establishes a multi-objective shortest path mixed-integer programming model based on a multimodal transportation network. To solve multi-objective shortest path problems with multimodal transportation, we design an ideal point method and propose a procedure for constructing the complete Pareto frontier based on the k-shortest path multi-objective algorithm. We use modified Dijkstra and Floyd multimodal transportation shortest path algorithms to build a k-shortest path multi-objective algorithm. The effectiveness of the proposed multimodal transportation shortest path algorithm is verified using empirical experiments carried out on test sets of different scales and a comparison of the runtime using a commercial solver. The results show that the modified Dijkstra algorithm has a runtime that is 100 times faster on average than the modified Floyd algorithm, which highlights its greater applicability in large-scale multimodal transportation networks, demonstrating that the proposed method both has practical significance and can generate satisfactory solutions to the multi-objective shortest path problem with multimodal transportation in the context of emergency logistics.

Improving Interpretability of Leucocyte Classification with Multimodal Network.

White blood cell classification plays a key role in the diagnosis of hematologic diseases. Models can perform classification either from images or based on morphological features. Image-based classification generally yields higher performance, but feature-based classification is more interpretable for clinicians. In this study, we employed a Multimodal neural network to classify white blood cells, utilizing a combination of images and morphological features. We compared this approach with image-only and feature-only training. While the highest performance was achieved with image-only training, the Multimodal model provided enhanced interpretability by the computation of SHAP values, and revealed crucial morphological features for biological characterization of the cells.

Multi-user low-upstream-loss multi-mode passive optical network with compact optical distribution network using multi-plane light conversion

Multi-user low-upstream-loss multi-mode passive optical network with compact optical distribution network using multi-plane light conversion

SGFNet: A semantic graph-based multimodal network for financial invoice information extraction

To meet the demand for a large amount of invoice entry work in the financial industry and improve the low accuracy of traditional manual entry, we construct SGFNet, a financial invoice information extraction network that integrates semantic graph associations and multimodal modeling. First, we construct a graph of strong and weak semantic associations between data within each modality based on the correlation of text content. Subsequently, we model the multimodal data in a unified structure, extract the text modal information of invoices along with corresponding image and layout modal information, and guide the fusion and embedding of multimodal data through semantic associations in the graph to produce a richer feature representation. Furthermore, semantically linked multimodal information is fed into an aggregated multimodal self-attention mechanism to establish effective connection between modalities. Finally, with the combination of supervised contrastive learning and smoothed Kullback–Leibler divergence in terms of loss functions, the accuracy degradation problem incurred by sample imbalance and convergence instability is reduced. In our experiments, we achieved F1 scores of 93.71% for the English financial invoice dataset and 96.27% for the Chinese dataset, indicating that the proposed method successfully extracts feature information from different data modalities, thereby achieving satisfactory information extraction results.

The Zwicky Transient Facility Bright Transient Survey. III. BTSbot: Automated Identification and Follow-up of Bright Transients with Deep Learning

The Bright Transient Survey (BTS) aims to obtain a classification spectrum for all bright (m peak ≤ 18.5 mag) extragalactic transients found in the Zwicky Transient Facility (ZTF) public survey. BTS critically relies on visual inspection (“scanning”) to select targets for spectroscopic follow-up, which, while effective, has required a significant time investment over the past ∼5 yr of ZTF operations. We present BTSbot, a multimodal convolutional neural network, which provides a bright transient score to individual ZTF detections using their image data and 25 extracted features. BTSbot is able to eliminate the need for daily human scanning by automatically identifying and requesting spectroscopic follow-up observations of new bright transient candidates. BTSbot recovers all bright transients in our test split and performs on par with scanners in terms of identification speed (on average, ∼1 hr quicker than scanners). We also find that BTSbot is not significantly impacted by any data shift by comparing performance across a concealed test split and a sample of very recent BTS candidates. BTSbot has been integrated into Fritz and Kowalski, ZTF’s first-party marshal and alert broker, and now sends automatic spectroscopic follow-up requests for the new transients it identifies. Between 2023 December and 2024 May, BTSbot selected 609 sources in real time, 96% of which were real extragalactic transients. With BTSbot and other automation tools, the BTS workflow has produced the first fully automatic end-to-end discovery and classification of a transient, representing a significant reduction in the human time needed to scan.

Multimodal Recipe Recommendation with Heterogeneous Graph Neural Networks

Recipe recommendation is the process of recommending suitable recipes to users based on factors such as user preferences and dietary needs. Recipes typically involve multiple modalities, with text and images being common, while most typical recipe recommendation methods recommend recipes to users based on text. Obviously, the expressiveness of a single modal is often not enough, and the semantic information of images is more abundant. Moreover, it is difficult to grasp the feature fusion granularity of different kinds of modal information and the relationship between recipes and users. To solve the above problem, this paper proposes a Multimodal Heterogeneous Graph Neural Network Recipe Recommendation (MHGRR) architecture, which aims to fully fuse the various kinds of modal information of recipes and handle the relationship between users and recipes. We use embedding and shallow Convolutional Neural Networks (CNNs) to extract original text and image information for unifying feature fusion granularity, and use Heterogeneous Graph Neural Networks based on GraphSAGE to capture the complex relationship between users and recipes. To verify the effectiveness of our proposed model, we perform some comparative experiments on a real dataset; the experiments show that our method outperforms most popular recipe recommendation methods. Through an ablation experiment, we found that adding image information to recipe recommendation is more effective, and we additionally found that as the output dimensions of GraphSAGE increased, the performance of the model varied little.

MGMI: A novel deep learning model based on short-term thermal load prediction

Accurate heat load prediction is the key to stable operation and control of district heating system. However, current heat load forecasting methods tend to treat individual buildings as isolated entities, ignoring the temporal and spatial correlation between buildings. In this paper, a heat load forecasting model is proposed which fully considers spatiotemporal information. Firstly, the spatial and temporal relationship between buildings is analyzed by statistical method. Secondly, synchronous wavelet transform (SWT) is used to denoise the heat load data to reduce the difficulty of prediction. Then, a hybrid model of multi-modal graph attention network (MG) and multi-scale Informer (MI) is constructed to capture spatiotemporal information in the data. Finally, an example of DHS in a campus in Liaoyang is analyzed. The results showed that compared with LSTM, CNN-Informer, GCN-LSTM and GAT-Informer, the MAE of MGMI on 168 h test set was reduced by 71.4%, 61.5%, 58% and 41.6%, respectively. And has the highest computational efficiency. The validity of the model is verified, which provides an important basis for the operation control of the thermal system.

CTDUNet: A Multimodal CNN-Transformer Dual U-Shaped Network with Coordinate Space Attention for Camellia oleifera Pests and Diseases Segmentation in Complex Environments.

Camellia oleifera is a crop of high economic value, yet it is particularly susceptible to various diseases and pests that significantly reduce its yield and quality. Consequently, the precise segmentation and classification of diseased Camellia leaves are vital for managing pests and diseases effectively. Deep learning exhibits significant advantages in the segmentation of plant diseases and pests, particularly in complex image processing and automated feature extraction. However, when employing single-modal models to segment Camellia oleifera diseases, three critical challenges arise: (A) lesions may closely resemble the colors of the complex background; (B) small sections of diseased leaves overlap; (C) the presence of multiple diseases on a single leaf. These factors considerably hinder segmentation accuracy. A novel multimodal model, CNN-Transformer Dual U-shaped Network (CTDUNet), based on a CNN-Transformer architecture, has been proposed to integrate image and text information. This model first utilizes text data to address the shortcomings of single-modal image features, enhancing its ability to distinguish lesions from environmental characteristics, even under conditions where they closely resemble one another. Additionally, we introduce Coordinate Space Attention (CSA), which focuses on the positional relationships between targets, thereby improving the segmentation of overlapping leaf edges. Furthermore, cross-attention (CA) is employed to align image and text features effectively, preserving local information and enhancing the perception and differentiation of various diseases. The CTDUNet model was evaluated on a self-made multimodal dataset compared against several models, including DeeplabV3+, UNet, PSPNet, Segformer, HrNet, and Language meets Vision Transformer (LViT). The experimental results demonstrate that CTDUNet achieved an mean Intersection over Union (mIoU) of 86.14%, surpassing both multimodal models and the best single-modal model by 3.91% and 5.84%, respectively. Additionally, CTDUNet exhibits high balance in the multi-class segmentation of Camellia oleifera diseases and pests. These results indicate the successful application of fused image and text multimodal information in the segmentation of Camellia disease, achieving outstanding performance.

Investigating the interaction between EEG and fNIRS: A multimodal network analysis of brain connectivity

The brain is a complex system with functional and structural networks. Different neuroimaging methods have been developed to explore these networks, but each method has its own unique strengths and limitations, depending on the signals they measure. Combining techniques like electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) has gained interest, but understanding how the information derived from these modalities is related to each other remains an exciting open question. The multilayer network model has emerged as a promising approach to integrate different sources data. In this study, we investigated the hemodynamic and electrophysiological data captured by fNIRS and EEG to compare brain network topologies derived from each modality, examining how these topologies vary between resting state (RS) and task-related conditions. Additionally, we adopted the multilayer network model to integrate EEG and fNIRS data and evaluate the benefits of combining multiple modalities compared to using a single modality in capturing characteristic brain functioning.A small-world network structure was observed in the rest, right motor imagery, and left motor imagery tasks in both modalities. We found that EEG captures faster changes in neural activity, thus providing a more precise estimation of the timing of information transfer between brain regions in RS. fNIRS provides insights into the slower hemodynamic responses associated with longer-lasting and sustained neural processes in cognitive tasks. The multilayer approach outperformed unimodal analyses, offering a richer understanding of brain function. Complementarity between EEG and fNIRS was observed, particularly during tasks, as well as a certain level of redundancy and complementarity between the multimodal and the unimodal approach, which depends on the modality and the specific brain state. Overall, the results highlight differences in how EEG and fNIRS capture brain network topology in RS and tasks and emphasize the value of integrating multiple modalities for a comprehensive view of brain connectivity and function.

Associations of alcohol and tobacco use with psychotic, depressive and developmental disorders revealed via multimodal neuroimaging

People affected by psychotic, depressive and developmental disorders are at a higher risk for alcohol and tobacco use. However, the further associations between alcohol/tobacco use and symptoms/cognition in these disorders remain unexplored. We identified multimodal brain networks involving alcohol use (n = 707) and tobacco use (n = 281) via supervised multimodal fusion and evaluated if these networks affected symptoms and cognition in people with psychotic (schizophrenia/schizoaffective disorder/bipolar, n = 178/134/143), depressive (major depressive disorder, n = 260) and developmental (autism spectrum disorder/attention deficit hyperactivity disorder, n = 421/346) disorders. Alcohol and tobacco use scores were used as references to guide functional and structural imaging fusion to identify alcohol/tobacco use associated multimodal patterns. Correlation analyses between the extracted brain features and symptoms or cognition were performed to evaluate the relationships between alcohol/tobacco use with symptoms/cognition in 6 psychiatric disorders. Results showed that (1) the default mode network (DMN) and salience network (SN) were associated with alcohol use, whereas the DMN and fronto-limbic network (FLN) were associated with tobacco use; (2) the DMN and fronto-basal ganglia (FBG) related to alcohol/tobacco use were correlated with symptom and cognition in psychosis; (3) the middle temporal cortex related to alcohol/tobacco use was associated with cognition in depression; (4) the DMN related to alcohol/tobacco use was related to symptom, whereas the SN and limbic system (LB) were related to cognition in developmental disorders. In summary, alcohol and tobacco use were associated with structural and functional abnormalities in DMN, SN and FLN and had significant associations with cognition and symptoms in psychotic, depressive and developmental disorders likely via different brain networks. Further understanding of these relationships may assist clinicians in the development of future approaches to improve symptoms and cognition among psychotic, depressive and developmental disorders.

MOSBY enables multi-omic inference and spatial biomarker discovery from whole slide images

The utility of deep neural nets has been demonstrated for mapping hematoxylin-and-eosin (H&E) stained image features to expression of individual genes. However, these models have not been employed to discover clinically relevant spatial biomarkers. Here we develop MOSBY (Multi-Omic translation of whole slide images for Spatial Biomarker discoverY) that leverages contrastive self-supervised pretraining to extract improved H&E whole slide images features, learns a mapping between image and bulk omic profiles (RNA, DNA, and protein), and utilizes tile-level information to discover spatial biomarkers. We validate MOSBY gene and gene set predictions with spatial transcriptomic and serially-sectioned CD8 IHC image data. We demonstrate that MOSBY-inferred colocalization features have survival-predictive power orthogonal to gene expression, and enable concordance indices highly competitive with survival-trained multimodal networks. We identify and validate (1) an ER stress-associated colocalization feature as a chemotherapy-specific risk factor in lung adenocarcinoma, and (2) the colocalization of T effector cell vs cysteine signatures as a negative prognostic factor in multiple cancer indications. The discovery of clinically relevant biologically interpretable spatial biomarkers showcases the utility of the model in unraveling novel insights in cancer biology as well as informing clinical decision-making.

M-GAN: multiattribute learning and multimodal feature fusion-based generative adversarial network for text-to-image synthesis

M-GAN: multiattribute learning and multimodal feature fusion-based generative adversarial network for text-to-image synthesis

M2DCapsN: Multimodal, Multichannel, and Dual-Step Capsule Network for Natural Language Moment Localization.

Natural language moment localization aims to localize the target moment that matches a given natural language query in an untrimmed video. The key to this challenging task is to capture fine-grained video-language correlations to establish the alignment between the query and target moment. Most existing works establish a single-pass interaction schema to capture correlations between queries and moments. Considering the complex feature space of lengthy video and diverse information between frames, the weight distribution of information interaction flow is prone to dispersion or misalignment, which leads to redundant information flow affecting the final prediction. We address this issue by proposing a capsule-based approach to model the query-video interactions, termed the Multimodal, Multichannel, and Dual-step Capsule Network ( [Formula: see text]DCapsN), which is derived from the intuition that "multiple people viewing multiple times is better than one person viewing one time." First, we introduce a multimodal capsule network, replacing the single-pass interaction schema of "one person viewing one time" with the iterative interaction schema of "one person viewing multiple times," which cyclically updates cross-modal interactions and modifies potential redundant interactions via its routing-by-agreement. Then, considering that the conventional routing mechanism only learns a single iterative interaction schema, we further propose a multichannel dynamic routing mechanism to learn multiple iterative interaction schemas, where each channel performs independent routing iteration to collectively capture cross-modal correlations from multiple subspaces, that is, "multiple people viewing." Moreover, we design a dual-step capsule network structure based on the multimodal, multichannel capsule network, bringing together the query and query-guided key moments to jointly enhance the original video, so as to select the target moments according to the enhanced part. Experimental results on three public datasets demonstrate the superiority of our approach in comparison with state-of-the-art methods, and comprehensive ablation and visualization analysis validate the effectiveness of each component of the proposed model.