Multimodal Convolutional Neural Network Research Articles

Multimodal Convolutional Neural Networks for Sperm Motility and Concentration Predictions

Semen analysis is an important analysis for male infertility primary investigation and manual semen analysis is a conventional method to assess it. Manual semen analysis has been revealed with accuracy and precision limitations due to noncompliance to guidelines and procedures. Sperm motility and concentration are the main indicators for pregnancy and conception rate hence they were selected for parameters prediction. Convolutional neural network (CNN) has benefited computer vision application industry in recent years and has been widely applied in computer vision research tasks. In this paper, three-dimensional CNN (3DCNN) was designed to extract motion and temporal features, which are vital for sperm motility prediction. For sperm concentration, since two-dimensional CNN (2DCNN) is efficient in recognizing and extracting spatial features, well-established Residual Network (ResNet) architecture was adopted and customized for sperm concentration prediction. Multimodal learning approach is a technique to aggregate learnt features from different deep learning architecture that adopted other forms of modalities, which could provide deep learning model with better insights on their tasks. Hence, a multimodal learning deep learning architecture was designed to receive both image-based (frames extracted from video samples) and video-based (stacked frames pre-processed from video samples) input that could provide well-extracted spatial and temporal features for sperm parameters prediction.  The results obtained using the proposed methodology have surpassed other similar research works who used deep learning approach. For sperm motility, its best achieved average mean absolute error (MAE) was 8.048, and sperm concentration obtained a competent Pearson’s correlation coefficient (RP) value of 0.853.

Automatic assessment of disproportionately enlarged subarachnoid-space hydrocephalus from 3D MRI using two deep learning models.

Disproportionately enlarged subarachnoid-space hydrocephalus (DESH) is a key feature for Hakim disease (idiopathic normal pressure hydrocephalus: iNPH), but subjectively evaluated. To develop automatic quantitative assessment of DESH with automatic segmentation using combined deep learning models. This study included 180 participants (42 Hakim patients, 138 healthy volunteers; 78 males, 102 females). Overall, 159 three-dimensional (3D) T1-weighted and 180 T2-weighted MRIs were included. As a semantic segmentation, 3D MRIs were automatically segmented in the total ventricles, total subarachnoid space (SAS), high-convexity SAS, and Sylvian fissure and basal cistern on the 3D U-Net model. As an image classification, DESH, ventricular dilatation (VD), tightened sulci in the high convexities (THC), and Sylvian fissure dilatation (SFD) were automatically assessed on the multimodal convolutional neural network (CNN) model. For both deep learning models, 110 T1- and 130 T2-weighted MRIs were used for training, 30 T1- and 30 T2-weighted MRIs for internal validation, and the remaining 19 T1- and 20 T2-weighted MRIs for external validation. Dice score was calculated as (overlapping area) × 2/total area. Automatic region extraction from 3D T1- and T2-weighted MRI was accurate for the total ventricles (mean Dice scores: 0.85 and 0.83), Sylvian fissure and basal cistern (0.70 and 0.69), and high-convexity SAS (0.68 and 0.60), respectively. Automatic determination of DESH, VD, THC, and SFD from the segmented regions on the multimodal CNN model was sufficiently reliable; all of the mean softmax probability scores were exceeded by 0.95. All of the areas under the receiver-operating characteristic curves of the DESH, Venthi, and Sylhi indexes calculated by the segmented regions for detecting DESH were exceeded by 0.97. Using 3D U-Net and a multimodal CNN, DESH was automatically detected with automatically segmented regions from 3D MRIs. Our developed diagnostic support tool can improve the precision of Hakim disease (iNPH) diagnosis.

A novel feature-level fusion scheme with multimodal attention CNN for heart sound classification

Background and objectiveMost of the existing machine learning-based heart sound classification methods achieve limited accuracy. Since they primarily depend on single domain feature information and tend to focus equally on each part of the signal rather than employing a selective attention mechanism. In addition, they fail to exploit convolutional neural network (CNN) - based features with an effective fusion strategy. MethodsIn order to overcome these limitations, a novel multimodal attention convolutional neural network (MACNN) with a feature-level fusion strategy, in which Mel-cepstral domain as well as general frequency domain features are incorporated to increase the diversity of the features, is proposed in this paper. In the proposed method, DilationAttenNet is first utilized to construct attention-based CNN feature extractors and then these feature extractors are jointly optimized in MACNN at the feature-level. The attention mechanism aims to suppress irrelevant information and focus on crucial diverse features extracted from the CNN. ResultsExtensive experiments are carried out to study the efficacy of the feature level fusion in comparison to that with early fusion. The results show that the proposed MACNN method significantly outperforms the state-of-the-art approaches in terms of accuracy and score for the two publicly available Github and Physionet datasets. ConclusionThe findings of our experiments demonstrated the high performance for heart sound classification based on the proposed MACNN, and hence have potential clinical usefulness in the identification of heart diseases. This technique can assist cardiologists and researchers in the design and development of heart sound classification methods.

Different gait combinations based on multi-modal deep CNN architectures

AbstractGait recognition is the process of identifying a person from a distance based on their walking patterns. However, the recognition rate drops significantly under cross-view angle and appearance-based variations. In this study, the effectiveness of the most well-known gait representations in solving this problem is investigated based on deep learning. For this purpose, a comprehensive performance evaluation is performed by combining different modalities, including silhouettes, optical flows, and concatenated image of the Gait Energy Image (GEI) head and leg region, with GEI itself. This evaluation is carried out across different multimodal deep convolutional neural network (CNN) architectures, namely fine-tuned EfficientNet-B0, MobileNet-V1, and ConvNeXt-base models. These models are trained separately on GEIs, silhouettes, optical flows, and concatenated image of GEI head and leg regions, and then extracted GEI features are fused in pairs with other extracted modality features to find the most effective gait combination. Experimental results on the two different datasets CASIA-B and Outdoor-Gait show that the concatenated image of GEI head and leg regions significantly increased the recognition rate of the networks compared to other modalities. Moreover, this modality demonstrates greater robustness under varied carrying (BG) and clothing (CL) conditions compared to optical flows (OF) and silhouettes (SF). Codes available at https://github.com/busrakckugurlu/Different-gait-combinations-based-on-multi-modal-deep-CNN-architectures.git

Multimodal CNN-DDI: using multimodal CNN for drug to drug interaction associated events.

Drug-to-drug interaction (DDIs) occurs when a patient consumes multiple drugs. Therefore, it is possible that any medication can influence other drugs' effectiveness. The drug-to-drug interactions are detected based on the interactions of chemical substructures, targets, pathways, and enzymes; therefore, machine learning (ML) and deep learning (DL) techniques are used to find the associated DDI events. The DL model, i.e., Convolutional Neural Network (CNN), is used to analyze the DDI. DDI is based on the 65 different drug-associated events, which is present in the drug bank database. Our model uses the inputs, which are chemical structures (i.e., smiles of drugs), enzymes, pathways, and the target of the drug. Therefore, for the multi-model CNN, we use several layers, activation functions, and features of drugs to achieve better accuracy as compared to traditional prediction algorithms. We perform different experiments on various hyperparameters. We have also carried out experiments on various iterations of drug features in different sets. Our Multi-Modal Convolutional Neural Network - Drug to Drug Interaction (MCNN-DDI) model achieved an accuracy of 90.00% and an AUPR of 94.78%. The results showed that a combination of the drug's features (i.e., chemical substructure, target, and enzyme) performs better in DDIs-associated events prediction than other features.

CSI-based WIFI Device Identification Solution

In recent years, wireless devices such as WIFI routers have become increasingly common in various fields. Rogue access point (RAP) is one of the threats that has persisted in wireless LAN (WLAN) for many years and can cause varying degrees of property damage and privacy leaks. In response to these threats, we propose a new security mechanism, called Rdi, which uses environment-independent features extracted from channel state information (CSI) as fingerprints for device identification. We found that the phase errors between multiple antennas on a single MIMO-OFDM (Multiple Input Multiple Output-Orthogonal Frequency Division Multiplexing) transmitter are not the same. This phase offset is due to the I/Q imbalance and imperfect oscillator of each WiFi network card, and it will not change with factors such as environment and time. Therefore, we inferred and verified that there must be some relationship between the phase errors of multiple groups of antennas, that is, the relative phase error (RPE). In addition, RPE will also vary with different WiFi devices. Compared with some similar fingerprint detection methods in the past, the use of specific connections between group phase errors between antennas can better reveal the different attributes between devices, thereby enhancing the uniqueness of features. Therefore, we believe that RPE can be used as an effective fingerprint to detect RAP attacks. We conduct a large number of experimental demonstrations on this, and innovatively built a multi-modal convolutional neural network (CNN) model to perform efficient classification work for our solution. Experiments on 22 WiFi devices and various scenarios show that the detection rate of Rdi can reach more than 98% in both dynamic and static device states.

Designing a cognitive smart healthcare framework for seizure prediction using multimodal convolutional neural network

Designing a cognitive smart healthcare framework for seizure prediction using multimodal convolutional neural network

Prediction of osteoporosis using MRI and CT scans with unimodal and multimodal deep-learning models.

Osteoporosis is the systematic degeneration of the human skeleton, with consequences ranging from a reduced quality of life to mortality. Therefore, the prediction of osteoporosis reduces risks and supports patients in taking precautions. Deep-learning and specific models achieve highly accurate results using different imaging modalities. The primary purpose of this research was to develop unimodal and multimodal deep-learning-based diagnostic models to predict bone mineral loss of the lumbar vertebrae using magnetic resonance (MR) and computed tomography (CT) imaging. Patients who received both lumbar dual-energy X-ray absorptiometry (DEXA) and MRI (n = 120) or CT (n = 100) examinations were included in this study. Unimodal and multimodal convolutional neural networks (CNNs) with dual blocks were proposed to predict osteoporosis using lumbar vertebrae MR and CT examinations in separate and combined datasets. Bone mineral density values obtained by DEXA were used as reference data. The proposed models were compared with a CNN model and six benchmark pre-trained deep-learning models. The proposed unimodal model obtained 96.54%, 98.84%, and 96.76% balanced accuracy for MRI, CT, and combined datasets, respectively, while the multimodal model achieved 98.90% balanced accuracy in 5-fold cross-validation experiments. Furthermore, the models obtained 95.68%-97.91% accuracy with a hold-out validation dataset. In addition, comparative experiments demonstrated that the proposed models yielded superior results by providing more effective feature extraction in dual blocks to predict osteoporosis. This study demonstrated that osteoporosis was accurately predicted by the proposed models using both MR and CT images, and a multimodal approach improved the prediction of osteoporosis. With further research involving prospective studies with a larger number of patients, there may be an opportunity to implement these technologies into clinical practice.

Generalizing Upper Limb Force Modeling With Transfer Learning: A Multimodal Approach Using EMG and IMU for New Users and Conditions.

In the field of EMG-based force modeling, the ability to generalize models across individuals could play a significant role in its adoption across a range of applications, including assistive devices, robotic and rehabilitation devices. However, current studies have predominately focused on intra-subject modeling, largely neglecting the burden of end-user data acquisition. In this work, we propose the use of transfer learning (TL) to generalize force modeling to a new user by first establishing a baseline model trained using other users' data, and then adapting to the end-user using a small amount of new data (only 10% , 20% , and 40% of the new user data). Using a deep multimodal convolutional neural network, consisting of two CNN models, one with high-density (HD) EMG and one with motion data recorded by an Inertial Measurement Unit (IMU), our proposed TL technique significantly improved force modeling compared to leave-one-subject-out (LOSO) and even intra-subject scenarios. The TL approach increased the average R squared values of the force modeling task by 60.81%, 190.53%, and 199.79% compared to the LOSO case, and by 13.4%, 36.88%, and 45.51% compared to the intra-subject case for isotonic, isokinetic and dynamic conditions, respectively. These results show that it is possible to adapt to a new user with minimal data while improving performance significantly compared to the intra-subject scenario. We also show that TL can be used to generalize on a new experimental condition for a new user.

Prognosis prediction of high grade serous adenocarcinoma based on multi-modal convolution neural network

Prognosis prediction of high grade serous adenocarcinoma based on multi-modal convolution neural network

DeepCompoundNet: enhancing compound–protein interaction prediction with multimodal convolutional neural networks

Virtual screening has emerged as a valuable computational tool for predicting compound–protein interactions, offering a cost-effective and rapid approach to identifying potential candidate drug molecules. Current machine learning-based methods rely on molecular structures and their relationship in the network. The former utilizes information such as amino acid sequences and chemical structures, while the latter leverages interaction network data, such as protein–protein interactions, drug–disease interactions, and protein–disease interactions. However, there has been limited exploration of integrating molecular information with interaction networks. This study presents DeepCompoundNet, a deep learning-based model that integrates protein features, drug properties, and diverse interaction data to predict chemical–protein interactions. DeepCompoundNet outperforms state-of-the-art methods for compound–protein interaction prediction, as demonstrated through performance evaluations. Our findings highlight the complementary nature of multiple interaction data, extending beyond amino acid sequence homology and chemical structure similarity. Moreover, our model’s analysis confirms that DeepCompoundNet gets higher performance in predicting interactions between proteins and chemicals not observed in the training samples. Communicated by Ramaswamy H. Sarma

3-D Tactile-Based Object Recognition for Robot Hands Using Force-Sensitive and Bend Sensor Arrays

Tactile sensing is a particularly important and challenging task for a modern robot to safely manipulate objects, interact with humans in a shared space, and provide various services. This paper presents a 3D tactile glove for robots with the combination of a piezoresistive-based force sensor array (412 sensors) covering the full hand and a resistive bend sensor array (5 sensors) on the back of five fingers. Deep learning-based CNN (Convolutional Neural Network) methods using the designed tactile glove are proposed for object recognition. In the experiment for recognizing 15 objects with a dexterous robot hand, an average classification accuracy of 93.67% has been achieved. Comparison experiments with three other typical classifiers (the Quadratic SVM, Weighted KNN, and Bagged Trees) and our CNN methods show an average recognition accuracy of 91.67% with the 3D tactile glove, revealing an accuracy improvement of 4.17% over only using the force sensor array. We further apply our 3D tactile glove and the multi-modal CNN to identify three other objects and demonstrate their generalization ability of tactile object recognition with an average success accuracy of 78.33%. The proposed 3D tactile glove can be further used in human-robot interactions, the design of prosthetics and humanoid robots, and for improving the intelligence level in brain-computer collaborative systems.

Investigation of an efficient multi-modal convolutional neural network for multiple sclerosis lesion detection

In this study, an automated 2D machine learning approach for fast and precise segmentation of MS lesions from multi-modal magnetic resonance images (mmMRI) is presented. The method is based on an U-Net like convolutional neural network (CNN) for automated 2D slice-based-segmentation of brain MRI volumes. The individual modalities are encoded in separate downsampling branches without weight sharing, to leverage the specific features. Skip connections input feature maps to multi-scale feature fusion (MSFF) blocks at every decoder stage of the network. Those are followed by multi-scale feature upsampling (MSFU) blocks which use the information about lesion shape and location. The CNN is evaluated on two publicly available datasets: The ISBI 2015 longitudinal MS lesion segmentation challenge dataset containing 19 subjects and the MICCAI 2016 MSSEG challenge dataset containing 15 subjects from various scanners. The proposed multi-input 2D architecture is among the top performing approaches in the ISBI challenge, to which open-access papers are available, is able to outperform state-of-the-art 3D approaches without additional post-processing, can be adapted to other scanners quickly, is robust against scanner variability and can be deployed for inference even on a standard laptop without a dedicated GPU.

SinkholeNet: A novel RGB-slope sinkhole dataset and deep weakly-supervised learning framework for sinkhole classification and localization

This paper proposes a novel multimodal deep weakly-supervised learning framework, SinkholeNet, to classify and localize sinkhole(s) in high-resolution RGB-slope aerial images. The SinkholeNet first employs a multimodal Convolutional Neural Network (CNN) architecture that simultaneously extracts features from the input RGB image and ground slope map and then fuses the extracted features. It then uses an improved ShuffleNet architecture on the fused features to classify patches as sinkholes or non-sinkholes. Finally, the last extracted feature maps, belonging to the sinkhole class, are used as input of gradient-weighted class activation mapping (Grad-CAM) to localize sinkhole(s) in a weakly-supervised setting. The proposed weakly-supervised framework intends to increase the available labeled data for training and decrease the cost of human annotation. We also introduce a novel publicly available weakly labeled sinkhole dataset comprising RGB-slope paired image patches to support reproducible research. The experimental results on the newly introduced dataset show that the SinkholeNet outperforms the other methods considered in this paper both for sinkhole classification and localization.

Detection of Pedestrians in Reverse Camera Using Multimodal Convolutional Neural Networks.

In recent years, the application of artificial intelligence (AI) in the automotive industry has led to the development of intelligent systems focused on road safety, aiming to improve protection for drivers and pedestrians worldwide to reduce the number of accidents yearly. One of the most critical functions of these systems is pedestrian detection, as it is crucial for the safety of everyone involved in road traffic. However, pedestrian detection goes beyond the front of the vehicle; it is also essential to consider the vehicle's rear since pedestrian collisions occur when the car is in reverse drive. To contribute to the solution of this problem, this research proposes a model based on convolutional neural networks (CNN) using a proposed one-dimensional architecture and the Inception V3 architecture to fuse the information from the backup camera and the distance measured by the ultrasonic sensors, to detect pedestrians when the vehicle is reversing. In addition, specific data collection was performed to build a database for the research. The proposed model showed outstanding results with 99.85% accuracy and 99.86% correct classification performance, demonstrating that it is possible to achieve the goal of pedestrian detection using CNN by fusing two types of data.

Multimodal Facial Emotion Recognition Using Improved Convolution Neural Networks Model

In the quest for human-robot interaction (HRI), leading to the development of emotion recognition, learning, and analysis capabilities, robotics plays a significant role in human perception, attention, decision-making, and social communication. However, the accurate recognition of emotions in HRI remains a challenge. This is due to the coexistence of multiple sources of information in utilizing multimodal facial expressions and head poses as multiple convolutional neural networks (CNN) and deep learning are combined. This research analyzes and improves the robustness of emotion recognition, and proposes a novel approach that optimizes traditional deep neural networks that fall into poor local optima when optimizing the weightings of the deep neural network using standard methods. The proposed approach adaptively finds the better weightings of the network, resulting in a hybrid genetic algorithm with stochastic gradient descent (HGASGD). This hybrid algorithm combines the inherent, implicit parallelism of the genetic algorithm with the better global optimization of stochastic gradient descent (SGD). An experiment shows the effectiveness of our proposed approach in providing complete emotion recognition through a combination of multimodal data, CNNs, and HGASGD, indicating that it represents a powerful tool in achieving interactions between humans and robotics. To validate and test the effectiveness of our proposed approach through experiments, the performance and reliability of our approach and two variants of HGASGD FER are compared using a large dataset of facial images. Our approach integrates multimodal information from facial expressions and head poses, enabling the system to recognize emotions better. The results show that CNN-HGASGD outperforms CNNs-SGD and other existing state-of-the-art methods in terms of FER.

The future of fish age estimation: deep machine learning coupled with Fourier transform near-infrared spectroscopy of otoliths

Our novel approach for fish age prediction uses quantitative analysis of Fourier transform near-infrared (FT-NIR) spectra of otoliths by means of multimodal convolutional neural networks (MMCNN). We integrate two key data modalities that are related to fish ages: the entire range of wavenumbers of FT-NIR spectra and corresponding biological and geospatial data for nearly 9000 walleye pollock ( Gadus chalcogrammus) specimens. The proposed model extracts informative spectral features automatically and elucidates hidden structural relationships associated with fish growth to improve age predictions. Absorbance associated with 7000 to 4000 cm−1 wavenumbers had the highest influence on model predictions followed by fish length, latitude, depth, and temperature. The optimal model resulted in good overall performance with an R2 of 0.93 and RMSE of 0.83 for training data set and R2 of 0.92 and RMSE of 0.91 for test data set. MMCNN's age predictions were comparable to microscope-based ages yielding as good or slightly better precision. Moreover, the model outperformed classical partial least squares analysis of otolith spectra and remedied prediction bias at older ages of fish.

A Convolutional Neural Network Using Multimodal Retinal Imaging for Differentiation of Mild Cognitive Impairment from Normal Cognition

To develop a machine learning tool capable of differentiating eyes of subjects with normal cognition from those with mild cognitive impairment (MCI) using OCT and OCT angiography (OCTA). Evaluation of a diagnostic technology. Subjects with normal cognition were compared to subjects with MCI. A multimodal convolutional neural network (CNN) was built to predict likelihood of MCI from ganglion cell-inner plexiform layer (GC-IPL) thickness maps, OCTA images, and quantitative data including patient characteristics. Area under the receiver operating characteristic curve (AUC) and summaries of the confusion matrix (sensitivity and specificity) were used as performance metrics for the prediction outputs of the CNN. Images from 236 eyes of 129 cognitively normal subjects and 154 eyes of 80 MCI subjects were used for training, validating, and testing the CNN. When applied to the independent test set using inputs including GC-IPL thickness maps, OCTA images, and quantitative OCT and OCTA data, the AUC value for the CNN was 0.809 (95% confidence interval [CI]: 0.681-0.937). This model achieved a sensitivity of 79% and specificity of 83%. The AUC value for GC-IPL thickness maps alone was 0.681 (95% CI: 0.529-0.832), for OCTA images alone was 0.625 (95% CI: 0.466-0.784) and for both GC-IPL maps and OCTA images was 0.693 (95% CI: 0.543-0.843). Models using quantitative data alone were also tested, with a model using quantitative data derived from images, 0.960 (95% CI: 0.902-1.00), outperforming a model using demographic data alone, 0.580 (95% CI: 0.417-0.742). This novel CNN was able to identify an MCI diagnosis using an independent test set comprised of OCT and OCTA images and quantitative data. The GC-IPL thickness maps provided more useful decision support than the OCTA images. The addition of quantitative data inputs also provided significant decision support to the CNN to identify individuals with MCI. Quantitative imaging metrics provided superior decision support than demographic data. Proprietary or commercial disclosure may be found after the references.

Multimodal convolutional neural networks for predicting evolution of gyrokinetic simulations

AbstractGyrokinetic simulations are required for the quantitative calculation of fluxes due to turbulence, which dominates over other transport mechanisms in tokamaks. However, nonlinear gyrokinetic simulations are computationally expensive. A multimodal convolutional neural network model that reads images and values generated by nonlinear gyrokinetic simulations and predicts electrostatic turbulent heat fluxes was developed to support efficient runs. The model was extended to account for squared electrostatic potential fluctuations, which are proportional to the fluxes in the quasilinear model, as well as images containing fluctuating electron and ion distribution functions and fluctuating electrostatic potentials in wavenumber space. This multimodal model can predict the time and electron and ion turbulent heat fluxes corresponding to the input data. The model trained on the Cyclone base case data successfully predicted times and fluxes not only for its test data, but also for the completely different and unknown JT‐60U case, with high accuracy. The predictive performance of the model depended on the similarity of the linear stability of the case used to train the model to the case being predicted.

Development and clinical validation of Swaasa AI platform for screening and prioritization of pulmonary TB

Acoustic signal analysis has been employed in various medical devices. However, studies involving cough sound analysis to screen the potential pulmonary tuberculosis (PTB) suspects are very few. The main objective of this cross-sectional validation study was to develop and validate the Swaasa AI platform to screen and prioritize at risk patients for PTB based on the signature cough sound as well as symptomatic information provided by the subjects. The voluntary cough sound data was collected at Andhra Medical College-India. An Algorithm based on multimodal convolutional neural network architecture and feedforward artificial neural network (tabular features) was built and validated on a total of 567 subjects, comprising 278 positive and 289 negative PTB cases. The output from these two models was combined to detect the likely presence (positive cases) of PTB. In the clinical validation phase, the AI-model was found to be 86.82% accurate in detecting the likely presence of PTB with 90.36% sensitivity and 84.67% specificity. The pilot testing of model was conducted at a peripheral health care centre, RHC Simhachalam-India on 65 presumptive PTB cases. Out of which, 15 subjects truly turned out to be PTB positive with a positive predictive value of 75%. The validation results obtained from the model are quite encouraging. This platform has the potential to fulfil the unmet need of a cost-effective PTB screening method. It works remotely, presents instantaneous results, and does not require a highly trained operator. Therefore, it could be implemented in various inaccessible, resource-poor parts of the world.