Multimodal Convolutional Neural Network Research Articles

Multimodal Convolutional Neural Network-Based Algorithm for Real-Time Detection and Differentiation of Malignant and Inflammatory Biliary Strictures in Cholangioscopy: a Proof of Concept Study with Video

Deep learning algorithms gained attention for detection (CADe) of biliary tract cancer (BTC) in digital single-operator cholangioscopy (dSOC). We developed a multimodal convolutional neural network (CNN) for detection (CADe) characterization and discriminating (CADx) between malignant, inflammatory and normal biliary tissue in raw dSOC videos. In addition, clinical metadata was included in the CNN algorithm to overcome limitations of image-only models. Based on dSOC videos and images of 111 patients (total of 15,158 still frames), we developed and validated a real-time CNN-based algorithm for CADe and CADx. We established an image-only model and metadata injection approach. In addition, we validated frame-wise and case-based predictions on complete dSOC video sequences. Model embeddings were visualized and class-activation maps highlighted relevant image regions. The concatenation-based CADx approach achieved a per-frame AUC of 0.871, sensitivity of 0.809 (95% CI: [0.784-0.832]), specificity of 0.773 [0.761-0.785], PPV of 0.450 [0.423-0.467], and NPV of 0.946 [0.940-0.954] with respect to malignancy on 5,715 test frames from complete videos of 20 patients. For case-based diagnosis using average prediction scores, six out of eight malignant cases and all twelve benign cases were identified correctly. Our algorithm distinguishes malignant and inflammatory bile duct lesions in dSOC videos, indicating the potential of CNN-based diagnostic support systems for both, CADe and CADx. The integration of non-image data can improve CNN based support systems, targeting current challenges in the assessment of biliary strictures.

The Zwicky Transient Facility Bright Transient Survey. III. BTSbot: Automated Identification and Follow-up of Bright Transients with Deep Learning

The Bright Transient Survey (BTS) aims to obtain a classification spectrum for all bright (m peak ≤ 18.5 mag) extragalactic transients found in the Zwicky Transient Facility (ZTF) public survey. BTS critically relies on visual inspection (“scanning”) to select targets for spectroscopic follow-up, which, while effective, has required a significant time investment over the past ∼5 yr of ZTF operations. We present BTSbot, a multimodal convolutional neural network, which provides a bright transient score to individual ZTF detections using their image data and 25 extracted features. BTSbot is able to eliminate the need for daily human scanning by automatically identifying and requesting spectroscopic follow-up observations of new bright transient candidates. BTSbot recovers all bright transients in our test split and performs on par with scanners in terms of identification speed (on average, ∼1 hr quicker than scanners). We also find that BTSbot is not significantly impacted by any data shift by comparing performance across a concealed test split and a sample of very recent BTS candidates. BTSbot has been integrated into Fritz and Kowalski, ZTF’s first-party marshal and alert broker, and now sends automatic spectroscopic follow-up requests for the new transients it identifies. Between 2023 December and 2024 May, BTSbot selected 609 sources in real time, 96% of which were real extragalactic transients. With BTSbot and other automation tools, the BTS workflow has produced the first fully automatic end-to-end discovery and classification of a transient, representing a significant reduction in the human time needed to scan.

Optimizing glaucoma diagnosis using fundus and optical coherence tomography image fusion based on multi-modal convolutional neural network approach

A novel approach that combines segmented fundus images (FIs) and optical coherence tomography image (OCTIs) are presented here, by incorporating deep learning network (DLN) techniques, to address the imperative need for advanced diagnostic algorithms in detecting and classifying glaucoma. By combining these two images, glaucoma diagnoses are made to improve the accuracy with more reliability. Multi modal convolutional neural networks (MMCNNs) are proposed for automatically extracting discriminatory features from both segmented FIs and OCTIs, allowing for comprehensive ocular analysis. A significant improvement in glaucoma diagnosis is achieved through segmentation of both FIs and OCTIs, ensuring robustness generalization to diverse clinical scenarios, DLN models are trained on datasets encompassing a wide range of glaucoma cases. The integrated approach outperforms individual modalities in terms of early detection of glaucoma and accurate classification. This method demonstrates promising potential in early glaucoma detection due to its effectiveness. By combining segmented features from both FIs and OCTIs through MMCNNs, improved efficiency in diagnosing predominant ocular glaucoma disorder is achieved compared to existing methods. Within the scope of this research, GoogLeNet (GN) is applied to independently classify glaucoma (uni-modal) in segmented FIs and OCTIs, providing a basis for comparison with the evaluation of MMCNNs.

Prediction of hematoma expansion in spontaneous intracerebral hemorrhage using a multimodal neural network

Hematoma expansion occasionally occurs in patients with intracerebral hemorrhage (ICH), associating with poor outcome. Multimodal neural networks incorporating convolutional neural network (CNN) analysis of images and neural network analysis of tabular data are known to show promising results in prediction and classification tasks. We aimed to develop a reliable multimodal neural network model that comprehensively analyzes CT images and clinical variables to predict hematoma expansion. We retrospectively enrolled ICH patients at four hospitals between 2017 and 2021, assigning patients from three hospitals to the training and validation dataset and patients from one hospital to the test dataset. Admission CT images and clinical variables were collected. CT findings were evaluated by experts. Three types of models were developed and trained: (1) a CNN model analyzing CT images, (2) a multimodal CNN model analyzing CT images and clinical variables, and (3) a non-CNN model analyzing CT findings and clinical variables with machine learning. The models were evaluated on the test dataset, focusing first on sensitivity and second on area under the receiver operating curve (AUC). Two hundred seventy-three patients (median age, 71 years [59–79]; 159 men) in the training and validation dataset and 106 patients (median age, 70 years [62–82]; 63 men) in the test dataset were included. Sensitivity and AUC of a CNN model were 1.000 (95% confidence interval [CI] 0.768–1.000) and 0.755 (95% CI 0.704–0.807); those of a multimodal CNN model were 1.000 (95% CI 0.768–1.000) and 0.799 (95% CI 0.749–0.849); and those of a non-CNN model were 0.857 (95% CI 0.572–0.982) and 0.733 (95% CI 0.625–0.840). We developed a multimodal neural network model incorporating CNN analysis of CT images and neural network analysis of clinical variables to predict hematoma expansion in ICH. The model was externally validated and showed the best performance of all the models.

Revolutionizing Brain Tumor Analysis: A Fusion of ChatGPT and Multi-Modal CNN for Unprecedented Precision

In this study, we introduce an innovative approach to significantly enhance the precision and interpretability of brain tumor detection and segmentation. Our method ingeniously integrates the cutting-edge capabilities of the ChatGPT chatbot interface with a state-of-the-art multi-modal convolutional neural network (CNN). Tested rigorously on the BraTS dataset, our method showcases unprecedented performance, outperforming existing techniques in terms of both accuracy and efficiency, with an impressive Dice score of 0.89 for tumor segmentation. By seamlessly integrating ChatGPT, our model unveils deep-seated insights into the intricate decision-making processes, providing researchers and physicians with invaluable understanding and confidence in the results. This groundbreaking fusion holds immense promise, poised to revolutionize the landscape of medical imaging, with far-reaching implications for clinical practice and research. Our study exemplifies the transformative potential achieved through the synergistic combination of multi-modal CNNs and natural language processing, paving the way for remarkable advancements in brain tumor detection and segmentation.

Multimodal Convolutional Neural Networks for Sperm Motility and Concentration Predictions

Semen analysis is an important analysis for male infertility primary investigation and manual semen analysis is a conventional method to assess it. Manual semen analysis has been revealed with accuracy and precision limitations due to noncompliance to guidelines and procedures. Sperm motility and concentration are the main indicators for pregnancy and conception rate hence they were selected for parameters prediction. Convolutional neural network (CNN) has benefited computer vision application industry in recent years and has been widely applied in computer vision research tasks. In this paper, three-dimensional CNN (3DCNN) was designed to extract motion and temporal features, which are vital for sperm motility prediction. For sperm concentration, since two-dimensional CNN (2DCNN) is efficient in recognizing and extracting spatial features, well-established Residual Network (ResNet) architecture was adopted and customized for sperm concentration prediction. Multimodal learning approach is a technique to aggregate learnt features from different deep learning architecture that adopted other forms of modalities, which could provide deep learning model with better insights on their tasks. Hence, a multimodal learning deep learning architecture was designed to receive both image-based (frames extracted from video samples) and video-based (stacked frames pre-processed from video samples) input that could provide well-extracted spatial and temporal features for sperm parameters prediction. The results obtained using the proposed methodology have surpassed other similar research works who used deep learning approach. For sperm motility, its best achieved average mean absolute error (MAE) was 8.048, and sperm concentration obtained a competent Pearson’s correlation coefficient (RP) value of 0.853.

Automatic assessment of disproportionately enlarged subarachnoid-space hydrocephalus from 3D MRI using two deep learning models.

Disproportionately enlarged subarachnoid-space hydrocephalus (DESH) is a key feature for Hakim disease (idiopathic normal pressure hydrocephalus: iNPH), but subjectively evaluated. To develop automatic quantitative assessment of DESH with automatic segmentation using combined deep learning models. This study included 180 participants (42 Hakim patients, 138 healthy volunteers; 78 males, 102 females). Overall, 159 three-dimensional (3D) T1-weighted and 180 T2-weighted MRIs were included. As a semantic segmentation, 3D MRIs were automatically segmented in the total ventricles, total subarachnoid space (SAS), high-convexity SAS, and Sylvian fissure and basal cistern on the 3D U-Net model. As an image classification, DESH, ventricular dilatation (VD), tightened sulci in the high convexities (THC), and Sylvian fissure dilatation (SFD) were automatically assessed on the multimodal convolutional neural network (CNN) model. For both deep learning models, 110 T1- and 130 T2-weighted MRIs were used for training, 30 T1- and 30 T2-weighted MRIs for internal validation, and the remaining 19 T1- and 20 T2-weighted MRIs for external validation. Dice score was calculated as (overlapping area) × 2/total area. Automatic region extraction from 3D T1- and T2-weighted MRI was accurate for the total ventricles (mean Dice scores: 0.85 and 0.83), Sylvian fissure and basal cistern (0.70 and 0.69), and high-convexity SAS (0.68 and 0.60), respectively. Automatic determination of DESH, VD, THC, and SFD from the segmented regions on the multimodal CNN model was sufficiently reliable; all of the mean softmax probability scores were exceeded by 0.95. All of the areas under the receiver-operating characteristic curves of the DESH, Venthi, and Sylhi indexes calculated by the segmented regions for detecting DESH were exceeded by 0.97. Using 3D U-Net and a multimodal CNN, DESH was automatically detected with automatically segmented regions from 3D MRIs. Our developed diagnostic support tool can improve the precision of Hakim disease (iNPH) diagnosis.

A novel feature-level fusion scheme with multimodal attention CNN for heart sound classification

Background and objectiveMost of the existing machine learning-based heart sound classification methods achieve limited accuracy. Since they primarily depend on single domain feature information and tend to focus equally on each part of the signal rather than employing a selective attention mechanism. In addition, they fail to exploit convolutional neural network (CNN) - based features with an effective fusion strategy. MethodsIn order to overcome these limitations, a novel multimodal attention convolutional neural network (MACNN) with a feature-level fusion strategy, in which Mel-cepstral domain as well as general frequency domain features are incorporated to increase the diversity of the features, is proposed in this paper. In the proposed method, DilationAttenNet is first utilized to construct attention-based CNN feature extractors and then these feature extractors are jointly optimized in MACNN at the feature-level. The attention mechanism aims to suppress irrelevant information and focus on crucial diverse features extracted from the CNN. ResultsExtensive experiments are carried out to study the efficacy of the feature level fusion in comparison to that with early fusion. The results show that the proposed MACNN method significantly outperforms the state-of-the-art approaches in terms of accuracy and score for the two publicly available Github and Physionet datasets. ConclusionThe findings of our experiments demonstrated the high performance for heart sound classification based on the proposed MACNN, and hence have potential clinical usefulness in the identification of heart diseases. This technique can assist cardiologists and researchers in the design and development of heart sound classification methods.

Different gait combinations based on multi-modal deep CNN architectures

Gait recognition is the process of identifying a person from a distance based on their walking patterns. However, the recognition rate drops significantly under cross-view angle and appearance-based variations. In this study, the effectiveness of the most well-known gait representations in solving this problem is investigated based on deep learning. For this purpose, a comprehensive performance evaluation is performed by combining different modalities, including silhouettes, optical flows, and concatenated image of the Gait Energy Image (GEI) head and leg region, with GEI itself. This evaluation is carried out across different multimodal deep convolutional neural network (CNN) architectures, namely fine-tuned EfficientNet-B0, MobileNet-V1, and ConvNeXt-base models. These models are trained separately on GEIs, silhouettes, optical flows, and concatenated image of GEI head and leg regions, and then extracted GEI features are fused in pairs with other extracted modality features to find the most effective gait combination. Experimental results on the two different datasets CASIA-B and Outdoor-Gait show that the concatenated image of GEI head and leg regions significantly increased the recognition rate of the networks compared to other modalities. Moreover, this modality demonstrates greater robustness under varied carrying (BG) and clothing (CL) conditions compared to optical flows (OF) and silhouettes (SF). Codes available at https://github.com/busrakckugurlu/Different-gait-combinations-based-on-multi-modal-deep-CNN-architectures.git

Multimodal CNN-DDI: using multimodal CNN for drug to drug interaction associated events.

Drug-to-drug interaction (DDIs) occurs when a patient consumes multiple drugs. Therefore, it is possible that any medication can influence other drugs' effectiveness. The drug-to-drug interactions are detected based on the interactions of chemical substructures, targets, pathways, and enzymes; therefore, machine learning (ML) and deep learning (DL) techniques are used to find the associated DDI events. The DL model, i.e., Convolutional Neural Network (CNN), is used to analyze the DDI. DDI is based on the 65 different drug-associated events, which is present in the drug bank database. Our model uses the inputs, which are chemical structures (i.e., smiles of drugs), enzymes, pathways, and the target of the drug. Therefore, for the multi-model CNN, we use several layers, activation functions, and features of drugs to achieve better accuracy as compared to traditional prediction algorithms. We perform different experiments on various hyperparameters. We have also carried out experiments on various iterations of drug features in different sets. Our Multi-Modal Convolutional Neural Network - Drug to Drug Interaction (MCNN-DDI) model achieved an accuracy of 90.00% and an AUPR of 94.78%. The results showed that a combination of the drug's features (i.e., chemical substructure, target, and enzyme) performs better in DDIs-associated events prediction than other features.

CSI-based WIFI Device Identification Solution

In recent years, wireless devices such as WIFI routers have become increasingly common in various fields. Rogue access point (RAP) is one of the threats that has persisted in wireless LAN (WLAN) for many years and can cause varying degrees of property damage and privacy leaks. In response to these threats, we propose a new security mechanism, called Rdi, which uses environment-independent features extracted from channel state information (CSI) as fingerprints for device identification. We found that the phase errors between multiple antennas on a single MIMO-OFDM (Multiple Input Multiple Output-Orthogonal Frequency Division Multiplexing) transmitter are not the same. This phase offset is due to the I/Q imbalance and imperfect oscillator of each WiFi network card, and it will not change with factors such as environment and time. Therefore, we inferred and verified that there must be some relationship between the phase errors of multiple groups of antennas, that is, the relative phase error (RPE). In addition, RPE will also vary with different WiFi devices. Compared with some similar fingerprint detection methods in the past, the use of specific connections between group phase errors between antennas can better reveal the different attributes between devices, thereby enhancing the uniqueness of features. Therefore, we believe that RPE can be used as an effective fingerprint to detect RAP attacks. We conduct a large number of experimental demonstrations on this, and innovatively built a multi-modal convolutional neural network (CNN) model to perform efficient classification work for our solution. Experiments on 22 WiFi devices and various scenarios show that the detection rate of Rdi can reach more than 98% in both dynamic and static device states.

RETRACTED ARTICLE: Designing a cognitive smart healthcare framework for seizure prediction using multimodal convolutional neural network

RETRACTED ARTICLE: Designing a cognitive smart healthcare framework for seizure prediction using multimodal convolutional neural network

Prediction of osteoporosis using MRI and CT scans with unimodal and multimodal deep-learning models.

Osteoporosis is the systematic degeneration of the human skeleton, with consequences ranging from a reduced quality of life to mortality. Therefore, the prediction of osteoporosis reduces risks and supports patients in taking precautions. Deep-learning and specific models achieve highly accurate results using different imaging modalities. The primary purpose of this research was to develop unimodal and multimodal deep-learning-based diagnostic models to predict bone mineral loss of the lumbar vertebrae using magnetic resonance (MR) and computed tomography (CT) imaging. Patients who received both lumbar dual-energy X-ray absorptiometry (DEXA) and MRI (n = 120) or CT (n = 100) examinations were included in this study. Unimodal and multimodal convolutional neural networks (CNNs) with dual blocks were proposed to predict osteoporosis using lumbar vertebrae MR and CT examinations in separate and combined datasets. Bone mineral density values obtained by DEXA were used as reference data. The proposed models were compared with a CNN model and six benchmark pre-trained deep-learning models. The proposed unimodal model obtained 96.54%, 98.84%, and 96.76% balanced accuracy for MRI, CT, and combined datasets, respectively, while the multimodal model achieved 98.90% balanced accuracy in 5-fold cross-validation experiments. Furthermore, the models obtained 95.68%-97.91% accuracy with a hold-out validation dataset. In addition, comparative experiments demonstrated that the proposed models yielded superior results by providing more effective feature extraction in dual blocks to predict osteoporosis. This study demonstrated that osteoporosis was accurately predicted by the proposed models using both MR and CT images, and a multimodal approach improved the prediction of osteoporosis. With further research involving prospective studies with a larger number of patients, there may be an opportunity to implement these technologies into clinical practice.

Generalizing Upper Limb Force Modeling With Transfer Learning: A Multimodal Approach Using EMG and IMU for New Users and Conditions.

In the field of EMG-based force modeling, the ability to generalize models across individuals could play a significant role in its adoption across a range of applications, including assistive devices, robotic and rehabilitation devices. However, current studies have predominately focused on intra-subject modeling, largely neglecting the burden of end-user data acquisition. In this work, we propose the use of transfer learning (TL) to generalize force modeling to a new user by first establishing a baseline model trained using other users' data, and then adapting to the end-user using a small amount of new data (only 10% , 20% , and 40% of the new user data). Using a deep multimodal convolutional neural network, consisting of two CNN models, one with high-density (HD) EMG and one with motion data recorded by an Inertial Measurement Unit (IMU), our proposed TL technique significantly improved force modeling compared to leave-one-subject-out (LOSO) and even intra-subject scenarios. The TL approach increased the average R squared values of the force modeling task by 60.81%, 190.53%, and 199.79% compared to the LOSO case, and by 13.4%, 36.88%, and 45.51% compared to the intra-subject case for isotonic, isokinetic and dynamic conditions, respectively. These results show that it is possible to adapt to a new user with minimal data while improving performance significantly compared to the intra-subject scenario. We also show that TL can be used to generalize on a new experimental condition for a new user.

Prognosis prediction of high grade serous adenocarcinoma based on multi-modal convolution neural network

Prognosis prediction of high grade serous adenocarcinoma based on multi-modal convolution neural network

DeepCompoundNet: enhancing compound–protein interaction prediction with multimodal convolutional neural networks

Virtual screening has emerged as a valuable computational tool for predicting compound–protein interactions, offering a cost-effective and rapid approach to identifying potential candidate drug molecules. Current machine learning-based methods rely on molecular structures and their relationship in the network. The former utilizes information such as amino acid sequences and chemical structures, while the latter leverages interaction network data, such as protein–protein interactions, drug–disease interactions, and protein–disease interactions. However, there has been limited exploration of integrating molecular information with interaction networks. This study presents DeepCompoundNet, a deep learning-based model that integrates protein features, drug properties, and diverse interaction data to predict chemical–protein interactions. DeepCompoundNet outperforms state-of-the-art methods for compound–protein interaction prediction, as demonstrated through performance evaluations. Our findings highlight the complementary nature of multiple interaction data, extending beyond amino acid sequence homology and chemical structure similarity. Moreover, our model’s analysis confirms that DeepCompoundNet gets higher performance in predicting interactions between proteins and chemicals not observed in the training samples. Communicated by Ramaswamy H. Sarma

3-D Tactile-Based Object Recognition for Robot Hands Using Force-Sensitive and Bend Sensor Arrays

Tactile sensing is a particularly important and challenging task for a modern robot to safely manipulate objects, interact with humans in a shared space, and provide various services. This paper presents a 3D tactile glove for robots with the combination of a piezoresistive-based force sensor array (412 sensors) covering the full hand and a resistive bend sensor array (5 sensors) on the back of five fingers. Deep learning-based CNN (Convolutional Neural Network) methods using the designed tactile glove are proposed for object recognition. In the experiment for recognizing 15 objects with a dexterous robot hand, an average classification accuracy of 93.67% has been achieved. Comparison experiments with three other typical classifiers (the Quadratic SVM, Weighted KNN, and Bagged Trees) and our CNN methods show an average recognition accuracy of 91.67% with the 3D tactile glove, revealing an accuracy improvement of 4.17% over only using the force sensor array. We further apply our 3D tactile glove and the multi-modal CNN to identify three other objects and demonstrate their generalization ability of tactile object recognition with an average success accuracy of 78.33%. The proposed 3D tactile glove can be further used in human-robot interactions, the design of prosthetics and humanoid robots, and for improving the intelligence level in brain-computer collaborative systems.

Investigation of an efficient multi-modal convolutional neural network for multiple sclerosis lesion detection

In this study, an automated 2D machine learning approach for fast and precise segmentation of MS lesions from multi-modal magnetic resonance images (mmMRI) is presented. The method is based on an U-Net like convolutional neural network (CNN) for automated 2D slice-based-segmentation of brain MRI volumes. The individual modalities are encoded in separate downsampling branches without weight sharing, to leverage the specific features. Skip connections input feature maps to multi-scale feature fusion (MSFF) blocks at every decoder stage of the network. Those are followed by multi-scale feature upsampling (MSFU) blocks which use the information about lesion shape and location. The CNN is evaluated on two publicly available datasets: The ISBI 2015 longitudinal MS lesion segmentation challenge dataset containing 19 subjects and the MICCAI 2016 MSSEG challenge dataset containing 15 subjects from various scanners. The proposed multi-input 2D architecture is among the top performing approaches in the ISBI challenge, to which open-access papers are available, is able to outperform state-of-the-art 3D approaches without additional post-processing, can be adapted to other scanners quickly, is robust against scanner variability and can be deployed for inference even on a standard laptop without a dedicated GPU.

SinkholeNet: A novel RGB-slope sinkhole dataset and deep weakly-supervised learning framework for sinkhole classification and localization

This paper proposes a novel multimodal deep weakly-supervised learning framework, SinkholeNet, to classify and localize sinkhole(s) in high-resolution RGB-slope aerial images. The SinkholeNet first employs a multimodal Convolutional Neural Network (CNN) architecture that simultaneously extracts features from the input RGB image and ground slope map and then fuses the extracted features. It then uses an improved ShuffleNet architecture on the fused features to classify patches as sinkholes or non-sinkholes. Finally, the last extracted feature maps, belonging to the sinkhole class, are used as input of gradient-weighted class activation mapping (Grad-CAM) to localize sinkhole(s) in a weakly-supervised setting. The proposed weakly-supervised framework intends to increase the available labeled data for training and decrease the cost of human annotation. We also introduce a novel publicly available weakly labeled sinkhole dataset comprising RGB-slope paired image patches to support reproducible research. The experimental results on the newly introduced dataset show that the SinkholeNet outperforms the other methods considered in this paper both for sinkhole classification and localization.

Detection of Pedestrians in Reverse Camera Using Multimodal Convolutional Neural Networks.

In recent years, the application of artificial intelligence (AI) in the automotive industry has led to the development of intelligent systems focused on road safety, aiming to improve protection for drivers and pedestrians worldwide to reduce the number of accidents yearly. One of the most critical functions of these systems is pedestrian detection, as it is crucial for the safety of everyone involved in road traffic. However, pedestrian detection goes beyond the front of the vehicle; it is also essential to consider the vehicle's rear since pedestrian collisions occur when the car is in reverse drive. To contribute to the solution of this problem, this research proposes a model based on convolutional neural networks (CNN) using a proposed one-dimensional architecture and the Inception V3 architecture to fuse the information from the backup camera and the distance measured by the ultrasonic sensors, to detect pedestrians when the vehicle is reversing. In addition, specific data collection was performed to build a database for the research. The proposed model showed outstanding results with 99.85% accuracy and 99.86% correct classification performance, demonstrating that it is possible to achieve the goal of pedestrian detection using CNN by fusing two types of data.