Multimodal Network Research Articles

Multimodal sleep staging network based on obstructive sleep apnea

BackgroundAutomatic sleep staging is essential for assessing sleep quality and diagnosing sleep disorders. While previous research has achieved high classification performance, most current sleep staging networks have only been validated in healthy populations, ignoring the impact of Obstructive Sleep Apnea (OSA) on sleep stage classification. In addition, it remains challenging to effectively improve the fine-grained detection of polysomnography (PSG) and capture multi-scale transitions between sleep stages. Therefore, a more widely applicable network is needed for sleep staging.MethodsThis paper introduces MSDC-SSNet, a novel deep learning network for automatic sleep stage classification. MSDC-SSNet transforms two channels of electroencephalogram (EEG) and one channel of electrooculogram (EOG) signals into time-frequency representations to obtain feature sequences at different temporal and frequency scales. An improved Transformer encoder architecture ensures temporal consistency and effectively captures long-term dependencies in EEG and EOG signals. The Multi-Scale Feature Extraction Module (MFEM) employs convolutional layers with varying dilation rates to capture spatial patterns from fine to coarse granularity. It adaptively fuses the weights of features to enhance the robustness of the model. Finally, multiple channel data are integrated to address the heterogeneity between different modalities effectively and alleviate the impact of OSA on sleep stages.ResultsWe evaluated MSDC-SSNet on three public datasets and our collection of PSG records of 17 OSA patients. It achieved an accuracy of 80.4% on the OSA dataset. It also outperformed the state-of-the-art methods in terms of accuracy, F1 score, and Cohen's Kappa coefficient on the remaining three datasets.ConclusionThe MSDC-SSRNet multi-channel sleep staging architecture proposed in this study enhances widespread system applicability by supplementing inter-channel features. It employs multi-scale attention to extract transition rules between sleep stages and effectively integrates multimodal information. Our method address the limitations of single-channel approaches, enhancing interpretability for clinical applications.

Mapping multi-modal dynamic network activity during naturalistic music listening

Abstract The human brain is a complex, adaptive system capable of parsing complex stimuli and generating complex behaviour. Understanding how to model and interpret the dynamic relationship between brain, behaviour, and the environment will provide vital information on how the brain responds to real-world stimuli, develops and ages, and adapts to pathology. Modelling together numerous streams of dynamic data, however, presents sizable methodological challenges. In this paper, we present a novel workflow and sample interpretation of a data set incorporating brain, behavioural, and stimulus data from a music listening study. We use hidden Markov modelling (HMM) to extract state timeseries from continuous high-dimensional EEG and stimulus data, estimate timeseries variables consistent with HMM from continuous low-dimensional behavioural data, and model the multi-modal data together using partial least squares (PLS). We offer a sample interpretation of the results, including a discussion on the limitations of the currently available tools, and discuss future directions for dynamic multi-modal analysis focusing on naturalistic behaviours.

TGNet: tensor-based graph convolutional networks for multimodal brain network analysis

Multimodal brain network analysis enables a comprehensive understanding of neurological disorders by integrating information from multiple neuroimaging modalities. However, existing methods often struggle to effectively model the complex structures of multimodal brain networks. In this paper, we propose a novel tensor-based graph convolutional network (TGNet) framework that combines tensor decomposition with multi-layer GCNs to capture both the homogeneity and intricate graph structures of multimodal brain networks. We evaluate TGNet on four datasets—HIV, Bipolar Disorder (BP), and Parkinson’s Disease (PPMI), Alzheimer’s Disease (ADNI)—demonstrating that it significantly outperforms existing methods for disease classification tasks, particularly in scenarios with limited sample sizes. The robustness and effectiveness of TGNet highlight its potential for advancing multimodal brain network analysis. The code is available at https://github.com/rongzhou7/TGNet.

Mesothelin expression prediction in pancreatic cancer based on multimodal stochastic configuration networks.

Predicting tumor biomarkers with high precision is essential for improving the diagnostic accuracy and developing more effective treatment strategies. This paper proposes a machine learning model that utilizes CT images and biopsy whole slide images (WSI) to classify mesothelin expression levels in pancreatic cancer. By combining multimodal learning and stochastic configuration networks, a radiopathomics mesothelin-prediction system named RPMSNet is developed. The system extracts radiomic and pathomic features from CT images and WSI, respectively, and sends them into stochastic configuration networks for the final prediction. Compared to traditional radiomics or pathomics, this system has the capability to capture more comprehensive image features, providing a multidimensional insight into tissue characteristics. The experiments and analyses demonstrate the accuracy and effectiveness of the system, with an area under the curve of 81.03%, an accuracy of 73.67%, a sensitivity of 71.54%, a precision of 76.78%, and a F1-score of 72.61%, surpassing both single-modality and dual-modality models. RPMSNet highlights its potential for early diagnosis and personalized treatment in precision medicine.

Multi-modality boundary-guided network for generalizable image manipulation localization

Multi-modality boundary-guided network for generalizable image manipulation localization

A multi-scale and multi-modal convolutional neural network for condition monitoring of transmission line

Abstract Monitoring the fatigue damage of transmission lines is crucial for stable power system operation. However, existing model-driven methods face challenges such as high computational complexity and reliance on expert knowledge, while data-driven methods require large amounts of abnormal state data. To address these issues, a multi-scale and multi-modal convolutional neural network (CNN) is proposed for real-time condition monitoring of transmission lines. Key steps include: firstly, empirical Fourier decomposition is used to decompose the original signals, extracting multi-scale state information at different frequency scales. Then, time-domain, frequency-domain, and time–frequency domain analyses are performed on the decomposed signals to capture multi-modal information. Based on this, a multi-modal fusion network is proposed based on a CNN to extract shallow and deep features, with a fully connected layer used for multi-modal feature fusion. Notably, the algorithm is implemented on a microprocessor for practical application. Experimental results show that the proposed model achieves a diagnostic accuracy of 93.06%, outperforming classical networks. It also surpasses models trained solely on time, frequency, or time–frequency features by 25.18%, 21.8%, and 19.3%, respectively.

CLFNet: a multi-modal data fusion network for traffic sign extraction

Abstract When using image data for signage extraction, poor visibility conditions such as insufficient light, rainy days, and low light intensity are encountered, leading to low accuracy and poor boundary segmentation in vision-based detection methods. To address this problem, we propose a cross-modal latent feature fusion network for signage detection, which obtains rich boundary information by combining images with light detection and ranging depth images, thus compensating for the pseudo-boundary phenomenon that may occur when using a single RGB image segmentation. First, HRNet is utilized as the backbone network to extract the boundary information of the point cloud depth map and RGB image by introducing the boundary extraction module; Second, the sensitivity to the boundary is enhanced by applying the feature aggregation module to deeply fuse the extracted boundary information with the image features; Finally, boundary Intersection over Union (IOU) is introduced as an evaluation index. The results show that the method performs more superiorly compared to the mainstream RGBD network, with an improvement of 5.5% and 6.1% in IOU and boundary IOU, and an accuracy of 98.3% and 96.2%, respectively, relative to the baseline network.

A prediction model for the mechanical properties of SUS316 stainless steel ultrathin strip driven by multimodal data mixing

Constructing a mapping relationship among material preparation process, microstructure, and mechanical properties is a challenge in material research and development. In this work, a deep learning framework for multimodal data fusion is constructed that couples a multi-layer perceptron (MLP) and a residual neural network (ResNet) to predict mechanical properties of SUS316 stainless steel ultrathin strips. Specifically, the MLP branch is used to extract the rolling process data features, and the ResNet with the addition of a convolutional block attention module (CBAM) is used to extract the microstructure features. Six models are constructed for comparison under the comprehensive consideration of factors such as unimodal network, the multimodal network and input form of image samples. The results show that the multimodal data model fused with the ResNet and MLP after adding the CBAM using both rolling process data and four types of microstructure image data as model inputs has the most accurate prediction results. The R2, MAPE, RMSE and MAE are 0.998, 0.727, 4.440 and 3.359, respectively. In addition, the proposed model is used for predicting yield strength and elongation, and the results show that the R2 values of both models on the test set are greater than 0.980, fully confirming that the multimodal data model has high prediction accuracy and good generalizability.

Estimating Strawberry Weight for Grading by Picking Robot with Point Cloud Completion and Multimodal Fusion Network

Estimating Strawberry Weight for Grading by Picking Robot with Point Cloud Completion and Multimodal Fusion Network

Multimodal Fusion Network for Crack Segmentation with Modified U-Net and Transfer Learning–Based MobileNetV2

Multimodal Fusion Network for Crack Segmentation with Modified U-Net and Transfer Learning–Based MobileNetV2

A conflict-free multi-modal fusion network with spatial reinforcement transformers for brain tumor segmentation

Brain gliomas are a leading cause of cancer mortality worldwide. Existing glioma segmentation approaches using multi-modal inputs often rely on a simplistic approach of stacking images from all modalities, disregarding modality-specific features that could optimize diagnostic outcomes. This paper introduces STE-Net, a spatial reinforcement hybrid Transformer-based tri-branch multi-modal evidential fusion network designed for conflict-free brain tumor segmentation. STE-Net features two independent encoder–decoder branches that process distinct modality sets, along with an additional branch that integrates features through a cross-modal channel-wise fusion (CMCF) module. The encoder employs a spatial reinforcement hybrid Transformer (SRHT), which combines a Swin Transformer block and a modified convolution block to capture richer spatial information. At the output level, a conflict-free evidential fusion mechanism (CEFM) is developed, leveraging the Dempster-Shafer (D-S) evidence theory and a conflict-solving strategy within a complex network framework. This mechanism ensures balanced reliability among the three output heads and mitigates potential conflicts. Each output is treated as a node in the complex network, and its importance is reassessed through the computation of direct and indirect weights to prevent potential mutual conflicts. We evaluate STE-Net on three public datasets: BraTS2018, BraTS2019, and BraTS2021. Both qualitative and quantitative results demonstrate that STE-Net outperforms several state-of-the-art methods. Statistical analysis further confirms the strong correlation between predicted tumors and ground truth. The code for this project is available at https://github.com/whotwin/STE-Net.

Multimodal neural network for enhanced protein stability prediction by integration of contact scores and spatial maps

Multimodal neural network for enhanced protein stability prediction by integration of contact scores and spatial maps

Network-based method for assessing multi-modal transportation network vulnerability to cascading failures

Transportation systems are vulnerable to disruptive events such as natural disasters, industrial accidents, terrorist attacks, and climate change. Vulnerability assessment is necessary to understand the impacts of these disruptive events, identify underlying deficiencies within the network, and improve transportation system resilience. Multi-modal transportation networks are often interdependent and form a “system of systems”, which creates a susceptibility to cascading indirect impacts within the integrated transportation network. Accurately modeling these interdependencies typically requires a large amount of data, such as traffic flow and travel demand information for transportation systems, which may not be available or accessible. This paper proposes a network topology-based framework to conduct an interdependent transportation network vulnerability analysis by introducing an algorithm to simulate cascading failures across transport systems. The proposed framework estimates the vulnerability of the network with respect to a specific hazard, combining the network topology and the functional attributes of the transportation infrastructure components. A case study with real-world data is conducted to demonstrate the applicability of the framework to the Houston freight transportation network, and to understand the network performance under different scenarios. This study presents an alternate method that stakeholders may use to assess interdependent transportation network vulnerability when more detailed flow-based data is not available.

Vehicle Trajectory Prediction With Multimodal and Dynamics-Aware Interaction Neural Networks

Vehicle Trajectory Prediction With Multimodal and Dynamics-Aware Interaction Neural Networks

An Adaptive Multimodal Fusion 3D Object Detection Algorithm for Unmanned Systems in Adverse Weather

Unmanned systems encounter challenging weather conditions during obstacle removal tasks. Researching stable, real-time, and accurate environmental perception methods under such conditions is crucial. Cameras and LiDAR sensors provide different and complementary data. However, the integration of disparate data presents challenges such as feature mismatches and the fusion of sparse and dense information, which can degrade algorithmic performance. Adverse weather conditions, like rain and snow, introduce noise that further reduces perception accuracy. To address these issues, we propose a novel weather-adaptive bird’s-eye view multi-level co-attention fusion 3D object detection algorithm (BEV-MCAF). This algorithm employs an improved feature extraction network to obtain more effective features. A multimodal feature fusion module has been constructed with BEV image feature generation and a co-attention mechanism for better fusion effects. A multi-scale multimodal joint domain adversarial network (M2-DANet) is proposed to enhance adaptability to adverse weather conditions. The efficacy of BEV-MCAF has been validated on both the nuScenes and Ithaca365 datasets, confirming its robustness and good generalization capability in a variety of bad weather conditions. The findings indicate that our proposed algorithm performs better than the benchmark, showing improved adaptability to harsh weather conditions and enhancing the robustness of UVs, ensuring reliable perception under challenging conditions.

MGSAN: multimodal graph self-attention network for skeleton-based action recognition

Due to the emergence of graph convolutional networks (GCNs), the skeleton-based action recognition has achieved remarkable results. However, the current models for skeleton-based action analysis treat skeleton sequences as a series of graphs, aggregating features of the entire sequence by alternately extracting spatial and temporal features, i.e., using a 2D (spatial features) plus 1D (temporal features) approach for feature extraction. This undoubtedly overlooks the complex spatiotemporal fusion relationships between joints during motion, making it challenging for models to capture the connections between different temporal frames and joints. In this paper, we propose a Multimodal Graph Self-Attention Network (MGSAN), which combines GCNs with self-attention to model the spatiotemporal relationships between skeleton sequences. Firstly, we design graph self-attention (GSA) blocks to capture the intrinsic topology and long-term temporal dependencies between joints. Secondly, we propose a multi-scale spatio-temporal convolutional network for channel-wise topology modeling (CW-TCN) to model short-term smooth temporal information of joint movements. Finally, we propose a multimodal fusion strategy to fuse joint, joint movement, and bone flow, providing the model with a richer set of multimodal features to make better predictions. The proposed MGSAN achieves state-of-the-art performance on three large-scale skeleton-based action recognition datasets, with accuracy of 93.1% on NTU RGB+D 60 cross-subject benchmark, 90.3% on NTU RGB+D 120 cross-subject benchmark, and 97.0% on the NW-UCLA dataset. Code is available at https://github.com/lizaowo/MGSAN.

Multimodal separation and cross fusion network based on Raman spectroscopy and FTIR spectroscopy for diagnosis of thyroid malignant tumor metastasis

The diagnosis of cervical lymph node metastasis from thyroid cancer is an essential stage in the progression of thyroid cancer. The metastasis of cervical lymph nodes directly affects the prognosis and survival rate of patients. Therefore, timely and early diagnosis is crucial for effective treatment and can significantly improve patients’ survival rate and quality of life. Traditional diagnostic methods, such as ultrasonography and radionuclide scanning, have limitations, such as complex operations and high missed diagnosis rates. Raman spectroscopy and FTIR spectroscopy can well reflect the molecular information of samples, have characteristics such as sensitivity and specificity, and are simple to operate. They have been widely used in clinical research in recent years. With the development of intelligent medical diagnosis technology, medical data shows a multi-modal trend. Compared with single-modal data, multi-modal data fusion can achieve complementary information, provide more comprehensive and valuable diagnostic information, significantly enhance the richness of data features, and improve the modeling effect of the model, helping to achieve better results. Accurate disease diagnosis. Existing research mostly uses cascade processing, ignoring the important correlations between multi-modal data, and at the same time not making full use of the intra-modal relationships that are also beneficial to prediction. We developed a new multi-modal separation cross-fusion network (MSCNet) based on deep learning technology. This network fully captures the complementary information between and within modalities through the feature separation module and feature cross-fusion module and effectively integrates Raman spectrum and FTIR spectrum data to diagnose thyroid cancer cervical lymph node metastasis accurately. The test results on the serum vibrational spectrum data set of 99 cases of cervical lymph node metastasis showed that the accuracy and AUC of a single Raman spectrum reached 63.63% and 63.78% respectively, and the accuracy and AUC of a single FTIR spectrum reached 95.84% respectively and 96%. The accuracy and AUC of Raman spectroscopy combined with FTIR spectroscopy reached 97.95% and 98% respectively, which is better than existing diagnostic technology. The omics correlation verification obtained correlation pairs of 5 Raman frequency shifts and 84 infrared spectral bands. This study provides new ideas and methods for the early diagnosis of cervical lymph node metastasis of thyroid cancer.

Automatic Modulation Recognition Based on Multimodal Information Processing: A New Approach and Application

Automatic modulation recognition (AMR) has wide applications in the fields of wireless communications, radar systems, and intelligent sensor networks. The existing deep learning-based modulation recognition models often focus on temporal features while overlooking the interrelations and spatio-temporal relationships among different types of signals. To overcome these limitations, a hybrid neural network based on a multimodal parallel structure, called the multimodal parallel hybrid neural network (MPHNN), is proposed to improve the recognition accuracy. The algorithm first preprocesses the data by parallelly processing the multimodal forms of the modulated signals before inputting them into the network. Subsequently, by combining Convolutional Neural Networks (CNN) and Bidirectional Gated Recurrent Unit (Bi-GRU) models, the CNN is used to extract spatial features of the received signals, while the Bi-GRU transmits previous state information of the time series to the current state to capture temporal features. Finally, the Convolutional Block Attention Module (CBAM) and Multi-Head Self-Attention (MHSA) are introduced as two attention mechanisms to handle the temporal and spatial correlations of the signals through an attention fusion mechanism, achieving the calibration of the signal feature maps. The effectiveness of this method is validated using various datasets, with the experimental results demonstrating that the proposed approach can fully utilize the information of multimodal signals. The experimental results show that the recognition accuracy of MPHNN on multiple datasets reaches 93.1%, and it has lower computational complexity and fewer parameters than other models.

Multimodal Networks of Signs between Language, Body and World in Poetry, Music and Architecture in the City of Valencia (Spain)

This work will deal with the sign networks that are established between language and world in the work of semiotic different arts (poetry, music and architecture). The objective is to understand and analyze from Enaction theory and from the Semiotic sign of Peirce how the constant flow and ebb between social environment, body and language configure both language and the world. It will also be shown how the work of art constitutes an emergency in which language, body and social environment participate. To achieve these objectives, poetic texts by the Valencian author Ausiàs March and artistic works related to them will be analyzed. The work methodology includes sign analysis, the comparison of original texts with translations and the comparison of works of art of semiotic different kinds.

Research on target recognition of multispectral streak tube imaging lidar system using multimodal convolutional neural network

A new multispectral streak tube imaging lidar is proposed to address the shortcomings of the traditional streak tube imaging lidar which can only provide single wavelength reflection information of the target. The accuracy of the depth map is improved by fusing the reconstructed depth images. To implement the target classification task, a dataset based on depth images and intensity images was first built, which consists of 240 targets with a total of 20 classes. Then, a multimodal neural network model was designed to classify the targets based on the characteristics of the dataset. The target classification ability of three methods which are depth images, depth images + intensity images and depth images + 3 intensity images are compared. The experimental results show that the proposed method can effectively improve the target recognition accuracy, which is increased from 85.19% to 90.47%.