Multimodal Deep Learning Model Research Articles

SAMoSA

Despite advances in audio- and motion-based human activity recognition (HAR) systems, a practical, power-efficient, and privacy-sensitive activity recognition system has remained elusive. State-of-the-art activity recognition systems often require power-hungry and privacy-invasive audio data. This is especially challenging for resource-constrained wearables, such as smartwatches. To counter the need for an always-on audio-based activity classification system, we first make use of power and compute-optimized IMUs sampled at 50 Hz to act as a trigger for detecting activity events. Once detected, we use a multimodal deep learning model that augments the motion data with audio data captured on a smartwatch. We subsample this audio to rates ≤ 1 kHz, rendering spoken content unintelligible, while also reducing power consumption on mobile devices. Our multimodal deep learning model achieves a recognition accuracy of 92.2% across 26 daily activities in four indoor environments. Our findings show that subsampling audio from 16 kHz down to 1 kHz, in concert with motion data, does not result in a significant drop in inference accuracy. We also analyze the speech content intelligibility and power requirements of audio sampled at less than 1 kHz and demonstrate that our proposed approach can improve the practicality of human activity recognition systems.

Multimodal deep learning model on interim [18F]FDG PET/CT for predicting primary treatment failure in diffuse large B-cell lymphoma.

The prediction of primary treatment failure (PTF) is necessary for patients with diffuse large B-cell lymphoma (DLBCL) since it serves as a prominent means for improving front-line outcomes. Using interim 18F-fluoro-2-deoxyglucose ([18F]FDG) positron emission tomography/computed tomography (PET/CT) imaging data, we aimed to construct multimodal deep learning (MDL) models to predict possible PTF in low-risk DLBCL. Initially, 205 DLBCL patients undergoing interim [18F]FDG PET/CT scans and the front-line standard of care were included in the primary dataset for model development. Then, 44 other patients were included in the external dataset for generalization evaluation. Based on the powerful backbone of the Conv-LSTM network, we incorporated five different multimodal fusion strategies (pixel intermixing, separate channel, separate branch, quantitative weighting, and hybrid learning) to make full use of PET/CT features and built five corresponding MDL models. Moreover, we found the best model, that is, the hybrid learning model, and optimized it by integrating the contrastive training objective to further improve its prediction performance. The final model with contrastive objective optimization, named the contrastive hybrid learning model, performed best, with an accuracy of 91.22% and an area under the receiver operating characteristic curve (AUC) of 0.926, in the primary dataset. In the external dataset, its accuracy and AUC remained at 88.64% and 0.925, respectively, indicating its good generalization ability. The proposed model achieved good performance, validated the predictive value of interim PET/CT, and holds promise for directing individualized clinical treatment. • The proposed multimodal models achieved accurate prediction of primary treatment failure in DLBCL patients. • Using an appropriate feature-level fusion strategy can make the same class close to each other regardless of the modal heterogeneity of the data source domain and positively impact the prediction performance. • Deep learning validated the predictive value of interim PET/CT in a way that exceeded human capabilities.

DeepThy‐Net: A Multimodal Deep Learning Method for Predicting Cervical Lymph Node Metastasis in Papillary Thyroid Cancer

Papillary thyroid cancer (PTC) accounts for more than 80% of thyroid cancers, and ultrasound (US) imaging is the preferred method for the diagnosis of PTC. However, accurate prediction of different patterns of cervical lymph node metastasis (CLNM) in PTC continues to be a challenge. Herein, US images and clinical factors of PTC patients from three hospitals for more than 11 years are collected, and a multimodal deep learning model called DeepThy‐Net is then developed to predict different CLNM patterns. The proposed model not only uses the convolutional features extracted by deep learning but also integrates traditional clinical factors that are highly related to lymph node metastasis. Finally, the model is tested in two independent test sets, and the experimental results show that the area under curve (AUC) is between 0.870 and 0.905, indicating clinical applicability. The proposed method provides an important reference for the treatment and management of PTC. Moreover, for PTC cases involving an active surveillance strategy, the proposed method can serve as an important CLNM early warning tool.

Incorporating deep learning into capacitive images for smartphone user authentication

We incorporate deep learning techniques into capacitive images of body parts (ear, four fingers, and thumb) to improve the performance of user authentication in smartphones. Use of a capacitive touchscreen as an image sensor has several advantages, such as it is less sensitive to poor illumination conditions, occlusions, and pose variations. Also, it does not need an additional hardware like iris or fingerprint scanner. Use of capacitive images for user authentication is not new. However, the performance, specially, false reject rates (FRRs) of the state-of-the-art capacitive image-based systems are poor. In this paper, we focus on improving the performance and leverage deep learning. Deep learning techniques demonstrated spectacular performance in previous physical biometrics-based research. However, to our knowledge, effectiveness of deep learning is still unexplored in capacitive touchscreen-based user authentication. In order to bridge this research gap, we devise a multi-modal deep learning model, namely UASNet, and compare its performance with a large set of uni- and multi-modal baselines. Using the UASNet, we achieve an accuracy of 99.77%, an EER of 0.48%, and an FRR of 1.19% at FAR of 0.06%.

Deep learning predicts immune checkpoint inhibitor-related pneumonitis from pretreatment computed tomography images.

Immune checkpoint inhibitors (ICIs) have revolutionized the treatment of lung cancer, including both non-small cell lung cancer and small cell lung cancer. Despite the promising results of immunotherapies, ICI-related pneumonitis (ICIP) is a potentially fatal adverse event. Therefore, early detection of patients at risk for developing ICIP before the initiation of immunotherapy is critical for alleviating future complications with early interventions and improving treatment outcomes. In this study, we present the first reported work that explores the potential of deep learning to predict patients who are at risk for developing ICIP. To this end, we collected the pretreatment baseline CT images and clinical information of 24 patients who developed ICIP after immunotherapy and 24 control patients who did not. A multimodal deep learning model was constructed based on 3D CT images and clinical data. To enhance performance, we employed two-stage transfer learning by pre-training the model sequentially on a large natural image dataset and a large CT image dataset, as well as transfer learning. Extensive experiments were conducted to verify the effectiveness of the key components used in our method. Using five-fold cross-validation, our method accurately distinguished ICIP patients from non-ICIP patients, with area under the receiver operating characteristic curve of 0.918 and accuracy of 0.920. This study demonstrates the promising potential of deep learning to identify patients at risk for developing ICIP. The proposed deep learning model enables efficient risk stratification, close monitoring, and prompt management of ICIP, ultimately leading to better treatment outcomes.

A multimodal deep learning model for cardiac resynchronisation therapy response prediction.

We present a novel multimodal deep learning framework for cardiac resynchronisation therapy (CRT) response prediction from 2D echocardiography and cardiac magnetic resonance (CMR) data. The proposed method first uses the 'nnU-Net' segmentation model to extract segmentations of the heart over the full cardiac cycle from the two modalities. Next, a multimodal deep learning classifier is used for CRT response prediction, which combines the latent spaces of the segmentation models of the two modalities. At test time, this framework can be used with 2D echocardiography data only, whilst taking advantage of the implicit relationship between CMR and echocardiography features learnt from the model. We evaluate our pipeline on a cohort of 50 CRT patients for whom paired echocardiography/CMR data were available, and results show that the proposed multimodal classifier results in a statistically significant improvement in accuracy compared to the baseline approach that uses only 2D echocardiography data. The combination of multimodal data enables CRT response to be predicted with 77.38% accuracy (83.33% sensitivity and 71.43% specificity), which is comparable with the current state-of-the-art in machine learning-based CRT response prediction. Our work represents the first multimodal deep learning approach for CRT response prediction.

D-vlog: Multimodal Vlog Dataset for Depression Detection

Detecting depression based on non-verbal behaviors has received great attention. However, most prior work on detecting depression mainly focused on detecting depressed individuals in laboratory settings, which are difficult to be generalized in practice. In addition, little attention has been paid to analyzing the non-verbal behaviors of depressed individuals in the wild. Therefore, in this paper, we present a multimodal depression dataset, D-Vlog, which consists of 961 vlogs (i.e., around 160 hours) collected from YouTube, which can be utilized in developing depression detection models based on the non-verbal behavior of individuals in real-world scenario. We develop a multimodal deep learning model that uses acoustic and visual features extracted from collected data to detect depression. Our proposed model employs the cross-attention mechanism to effectively capture the relationship across acoustic and visual features, and generates useful multimodal representations for depression detection. The extensive experimental results demonstrate that the proposed model significantly outperforms other baseline models. We believe our dataset and the proposed model are useful for analyzing and detecting depressed individuals based on non-verbal behavior.

A multi-modal deep learning model integrates clinical, pathomic, and radiomic features for glioma classification and grading.

e14038 Background: Histopathologic glioma diagnosis relies on the interpretation of the morphologic appearance of tumor cells. This conventional approach suffers from documented inter-pathologist disagreements with minor discordances in 36% and serious discordances in 8.8% of cases (Bruner JM et al. Cancer. 1997). Machine learning applied to automatically extracted pathomic features offers the potential to reduce this inter-reader variability and improve classification accuracy. The addition of clinical history and radiomic features may further improve performance. This work compares the performance of automatic pathomic and radiomic feature analysis to human histologic impressions for glioma classification according to the 2021 WHO scheme. Methods: We developed and evaluated models for predicting the 2021 WHO Glioma class using data from The Cancer Genome Atlas and The Cancer Imaging Archive. We used a cohort of 112 glioma cases with complete data of molecular markers, digital slides, and MRI images. Using molecular markers, we divided this cohort into 2021 WHO types, including 61 glioblastomas, 38 astrocytomas, and 13 oligodendroglioma. We extracted 374 pathomic features summarizing the tumor-rich region on each digital slide. Neural network models were developed with pathomic features alone or in combination with clinical factors (sex, age, and Karnofsky index) and 670 radiomic features obtained from multi-sequence MRI exams. Accuracy was estimated with 5-fold cross validation. Feature importance analysis was carried out with recursive feature elimination to identify the top pathomic and radiomic features related to the specific glioma diagnosis. Results: In this cohort, the conventional histologic impression, reliant on pathologist interpretation, was successful in identifying the WHO glioma class with an accuracy of 0.77 +/- 0.08. The use of pathomics features alone, representing a fully automatic approach, had an accuracy of 0.83 +/- 0.08. The use of pathomics and radiomics information increased the accuracy to 0.88 +/- 0.08. Feature importance analysis revealed that the most important pathomic features were related to nuclear size, nuclear hematoxylin staining intensity, and nuclear circularity. Conclusions: In this pilot study, our machine learning model with pathomic features achieved promising performance for glioma classification. Combining clinical and radiomic features further enhanced performance. We found that the pathomic features that most correlated with the pathologic classification are the same key features that pathologists use in their diagnostic evaluation. While molecular testing is now the definitive reference information for final glioma classification, this pilot study suggests that pathomics analysis could have a valuable role during a pathologist’s initial histologic review.

Prostate cancer risk in African American men evaluated via digital histopathology multi-modal deep learning models developed on NRG Oncology phase III clinical trials.

108 Background: Artificial intelligence (AI) tools can display racial bias as a result of existing systemic health inequities and biased datasets. We have previously developed multi-modal AI (MMAI) prognostic models based on digital pathology images from five phase III randomized radiotherapy prostate cancer trials that outperform NCCN risk groups for prediction of distant metastasis (DM), biochemical failure (BF), prostate cancer-specific mortality (PCSM) and all-cause mortality (OS). In this study, we assessed the algorithmic fairness of the locked MMAI models between African American (AA) and non-AA populations in the five randomized trials. Methods: Patients enrolled in NRG/RTOG 9202, 9408, 9413, 9910, and 0126 with digitized biopsy histopathology slides were included in this study. The locked MMAI models were applied, and subgroup analyses were conducted by comparing distributions of clinical variables and MMAI scores (medians for continuous variables and proportions for categorical variables reported), and evaluating MMAI models’ prognostic ability among AA and non-AA men. The performance of the models were compared using DM as the primary endpoint and secondary endpoints of BF, PCSM, OS (death without an event as a competing risk) with Fine-Gray or Cox Proportional Hazards models. Either Kaplan Meier or cumulative incidence estimates were computed and compared using log-rank or Gray’s test. Results: This study included 5,624 men: 932 (17%) AA, 4503 (80%) white, and 189 (3%) other races. AA had younger median age (69 vs 71 year [yr]), higher median baseline PSA (12 vs 10 ng/mL), more T1-T2a (62% vs 57%), more Gleason < 7 (42% vs 36%) and 8-10 (15% vs 12%), and more NCCN low and high risk (12% vs 10% and 41% vs 33%). AA and non-AA had estimated 5-yr BF rates 27% and 27%, 5-yr DM rates 5% and 5%, 10-yr PCSM 5% and 7%, and 10-yr OS 58% and 60%, respectively. The median (interquartile range) score of the model optimizing for 5-yr DM (5-yr DM MMAI) was 0.044 (0.037–0.059) in AA and 0.043 (0.036–0.057) in non-AA. Similarly, all other MMAI models had differences in the medians between AA and non-AA ranging from 0.001 to 0.02. For all endpoints, the 5-yr DM MMAI model showed strong prognostic signal (hazard ratio [HR] per one standard deviation increase: 1.6 for DM, 1.4 for BF, 1.6 for PCSM and 1.3 for OS, all p-values < 0.001) and had comparable trends within AA vs. non-AA in the entire cohort (e.g., HR for DM 1.4 vs 1.6). Similar results were observed for the MMAI model optimizing for 10-yr PCSM. Conclusions: To our knowledge, this represents the first comparative analyses of a digital pathology AI prognostic model in AA vs. non-AA prostate cancer patients. The prognostic performance of the AI models was found to be comparable between subgroups. Our data supports the use of these models across racial groups, though further validation in AA cohorts is ongoing.

Emerging artificial intelligence methods for fighting lung cancer: A survey

Lung cancer has one of the highest incidence rates and mortality rates among all common cancers worldwide. Early detection of suspicious lung nodules is crucial in fighting lung cancer. In recent years, with the proliferation of clinical data like low-dose computed tomography (LDCT), histology whole slide images, electronic health records, and sensor readings from medical IoT devices etc., many artificial intelligence tools have taken more important roles in lung cancer management. In this survey, we lay out the current and emergent artificial intelligence methods for fighting lung cancers. Besides the commonly used CT image based deep learning models for detecting and diagnosing lung nodules, we also cover emergent AI techniques for lung cancer: 1) federated deep learning models for harnessing multi-center data with privacy in mind, 2) multi-modal deep learning models for integrating multiple sources of clinical and image data, 3) interpretable deep learning models for opening the black box for clinicians. In the big data era for cancer management, we believe this short survey will help AI researchers better understand the clinical challenges of lung cancer and will also help clinicians better understand the emergent AI tools.

Multimodal deep learning model for human handover classification

Giving and receiving objects between humans and robots is a critical task which collaborative robots must be able to do. In order for robots to achieve that, they must be able to classify different types of human handover motions. Previous works did not mainly focus on classifying the motion type from both giver and receiver perspectives. However, they solely focused on object grasping, handover detection, and handover classification from one side only (giver/receiver). This paper discusses the design and implementation of different deep learning architectures with long short term memory (LSTM) network; and different feature selection techniques for human handover classification from both giver and receiver perspectives. Classification performance while using unimodal and multimodal deep learning models is investigated. The data used for evaluation is a publicly available dataset with four different modalities: motion tracking sensors readings, Kinect readings for 15 joints positions, 6-axis inertial sensor readings, and video recordings. The multimodality added a huge boost in the classification performance; achieving 96% accuracy with the feature selection based deep learning architecture.

Multimodal Deep Learning Models for Detecting Dementia From Speech and Transcripts.

Alzheimer's dementia (AD) entails negative psychological, social, and economic consequences not only for the patients but also for their families, relatives, and society in general. Despite the significance of this phenomenon and the importance for an early diagnosis, there are still limitations. Specifically, the main limitation is pertinent to the way the modalities of speech and transcripts are combined in a single neural network. Existing research works add/concatenate the image and text representations, employ majority voting approaches or average the predictions after training many textual and speech models separately. To address these limitations, in this article we present some new methods to detect AD patients and predict the Mini-Mental State Examination (MMSE) scores in an end-to-end trainable manner consisting of a combination of BERT, Vision Transformer, Co-Attention, Multimodal Shifting Gate, and a variant of the self-attention mechanism. Specifically, we convert audio to Log-Mel spectrograms, their delta, and delta-delta (acceleration values). First, we pass each transcript and image through a BERT model and Vision Transformer, respectively, adding a co-attention layer at the top, which generates image and word attention simultaneously. Secondly, we propose an architecture, which integrates multimodal information to a BERT model via a Multimodal Shifting Gate. Finally, we introduce an approach to capture both the inter- and intra-modal interactions by concatenating the textual and visual representations and utilizing a self-attention mechanism, which includes a gate model. Experiments conducted on the ADReSS Challenge dataset indicate that our introduced models demonstrate valuable advantages over existing research initiatives achieving competitive results in both the AD classification and MMSE regression task. Specifically, our best performing model attains an accuracy of 90.00% and a Root Mean Squared Error (RMSE) of 3.61 in the AD classification task and MMSE regression task, respectively, achieving a new state-of-the-art performance in the MMSE regression task.

A Novel Deep Learning Method to Predict Lung Cancer Long-Term Survival With Biological Knowledge Incorporated Gene Expression Images and Clinical Data.

Lung cancer is the leading cause of the cancer deaths. Therefore, predicting the survival status of lung cancer patients is of great value. However, the existing methods mainly depend on statistical machine learning (ML) algorithms. Moreover, they are not appropriate for high-dimensionality genomics data, and deep learning (DL), with strong high-dimensional data learning capability, can be used to predict lung cancer survival using genomics data. The Cancer Genome Atlas (TCGA) is a great database that contains many kinds of genomics data for 33 cancer types. With this enormous amount of data, researchers can analyze key factors related to cancer therapy. This paper proposes a novel method to predict lung cancer long-term survival using gene expression data from TCGA. Firstly, we select the most relevant genes to the target problem by the supervised feature selection method called mutual information selector. Secondly, we propose a method to convert gene expression data into two kinds of images with KEGG BRITE and KEGG Pathway data incorporated, so that we could make good use of the convolutional neural network (CNN) model to learn high-level features. Afterwards, we design a CNN-based DL model and added two kinds of clinical data to improve the performance, so that we finally got a multimodal DL model. The generalized experiments results indicated that our method performed much better than the ML models and unimodal DL models. Furthermore, we conduct survival analysis and observe that our model could better divide the samples into high-risk and low-risk groups.

Multimodal deep learning applied to classify healthy and disease states of human microbiome

Metagenomic sequencing methods provide considerable genomic information regarding human microbiomes, enabling us to discover and understand microbial diseases. Compositional differences have been reported between patients and healthy people, which could be used in the diagnosis of patients. Despite significant progress in this regard, the accuracy of these tools needs to be improved for applications in diagnostics and therapeutics. MDL4Microbiome, the method developed herein, demonstrated high accuracy in predicting disease status by using various features from metagenome sequences and a multimodal deep learning model. We propose combining three different features, i.e., conventional taxonomic profiles, genome-level relative abundance, and metabolic functional characteristics, to enhance classification accuracy. This deep learning model enabled the construction of a classifier that combines these various modalities encoded in the human microbiome. We achieved accuracies of 0.98, 0.76, 0.84, and 0.97 for predicting patients with inflammatory bowel disease, type 2 diabetes, liver cirrhosis, and colorectal cancer, respectively; these are comparable or higher than classical machine learning methods. A deeper analysis was also performed on the resulting sets of selected features to understand the contribution of their different characteristics. MDL4Microbiome is a classifier with higher or comparable accuracy compared with other machine learning methods, which offers perspectives on feature generation with metagenome sequences in deep learning models and their advantages in the classification of host disease status.

CAMM: Cross-Attention Multimodal Classification of Disaster-Related Tweets

During the past decade, social media platforms have been extensively used during a disaster for information dissemination by the affected community and humanitarian agencies. Although many studies have been done recently to classify the informative and non-informative messages from social media posts, most are unimodal, i.e., have independently used textual or visual data to build the deep learning models. In the present study, we integrate the complementary information provided by the text and image messages about the same event posted by the affected community on the social media platform Twitter and build a multimodal deep learning model based on the concept of attention mechanism. The attention mechanism is a recent breakthrough that has revolutionized the field of deep learning. Just as humans pay more attention to a specific part of the text or image, ignoring the rest, neural networks can also be trained to concentrate on more relevant features through the attention mechanism. We propose a novel Cross-Attention Multi-Modal (CAMM) deep neural network for classifying multimodal disaster data, which uses the attention mask of the textual modality to highlight the features of the visual modality. We compare CAMM with unimodal models and the most popular bilinear multimodal models, MUTAN and BLOCK, generally used for visual question answering. CAMM achieves an average F1-score of 84.08%, better than the MUTAN and BLOCK methods by 6.31% and 5.91%, respectively. The proposed cross-attention-based multimodal deep learning method outperforms the current state-of-the-art fusion methods on the benchmark multimodal disaster dataset by highlighting the more relevant cross-domain features of text and image tweets. This study affirms that social media platforms become a rich source of multimodal data during a disaster. This data can be utilized to build automated tools for quick filtration of informative messages to assess the post-disaster needs of the affected community and provide timely help.

DeepNCI: DFT Noncovalent Interaction Correction with Transferable Multimodal Three-Dimensional Convolutional Neural Networks.

A multimodal deep learning model, DeepNCI, is proposed for improving noncovalent interactions (NCIs) calculated via density functional theory (DFT). DeepNCI is composed of a three-dimensional convolutional neural network (3D CNN) for abstracting critical and comprehensive features from 3D electron density, and a neural network for modeling one-dimensional quantum chemical properties. By merging features from two networks, DeepNCI is able to reduce the root-mean-square error of DFT-calculated NCI from 1.19 kcal/mol to ∼0.2 kcal/mol for a NCI molecular database (>1000 molecules). The representativeness of the joint features can be visualized by t-distributed stochastic neighbor embedding (t-SNE), where they can distinguish categorized NCI systems quite well. Therefore, the fused model performs better than its component networks. In addition, the 3D CNN takes electron density as inputs that are in the same range, despite the size of molecular systems, so it can promote model applicability and transferability. To clarify the applicability of DeepNCI, an application domain (AD) has been defined with merged features using the K-nearest-neighbor method. The calculations for external test sets are shown that AD can properly monitor the reliability for a prediction. The model transferability is tested with a small database of homolysis bond dissociation energy including only dozens of samples. With NCI database pretrained parameters, the same or better performance than the reported results is achieved by transfer learning. This suggests that the DeepNCI model is transferable and it may transfer to other relative tasks, which possibly can resolve some small sampling problems. The source code of DeepNCI can be freely accessed at https://github.com/wenzelee/DeepNCI.

Performance of deep learning in mapping water quality of Lake Simcoe with long-term Landsat archive

Remote sensing provides full-coverage and dynamic water quality monitoring with high efficiency and low consumption. Deep learning (DL) has been progressively used in water quality retrieval because it efficiently captures the potential relationship between target variables and imagery. In this study, the multimodal deep learning (MDL) models were developed and rigorously validated using atmospherically corrected Landsat remote sensing reflectance data and synchronous water quality measurements for estimating long-term Chlorophyll-a (Chl-a), total phosphorus (TP), and total nitrogen (TN) in Lake Simcoe, Canada. Since TP and TN are non-optically active, their retrievals were based on the fact that they are closely related to the optically active constituents (OACs) such as Chl-a. We trained the MDL models with one in-situ measured data set (for Chl-a, N = 315, for TP and TN, N = 303), validated the models with two independent data sets (N = 147), and compared the model performances with several DL, machine learning, and empirical algorithms. The results indicated that the MDL models adequately estimated Chl-a (mean absolute error (MAE) = 32.57%, Bias = 10.61%), TP (MAE = 42.58%, Bias = −2.82%), and TN (MAE = 35.05%, Bias = 13.66%), and outperformed several other candidate algorithms, namely the progressively decreasing deep neural network (DNN), a DNN with trainable parameters similar to MDL but without splitting input features, the eXtreme Gradient Boosting, the support vector regression, the NASA Ocean Color two-band and three-band ratio algorithms, and another empirical algorithm of Landsat data in clear lakes. Using the MDL models, we reconstructed the historical spatiotemporal patterns of Chl-a, TP, and TN in Lake Simcoe since 1984, and investigated the effects of two water quality improvement programs. In addition, the physical mechanism and interpretability of the MDL models were explored by quantifying the contribution of each feature to the model outputs. The framework proposed in this study provides a practical method for long-term Chl-a, TP, and TN estimation at the regional scale.

A whole-process interpretable and multi-modal deep reinforcement learning for diagnosis and analysis of Alzheimer’s disease ∗ ∗ Data used in preparation of this article were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu). As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or

Objective. Alzheimer’s disease (AD), a common disease of the elderly with unknown etiology, has been adversely affecting many people, especially with the aging of the population and the younger trend of this disease. Current artificial intelligence (AI) methods based on individual information or magnetic resonance imaging (MRI) can solve the problem of diagnostic sensitivity and specificity, but still face the challenges of interpretability and clinical feasibility. In this study, we propose an interpretable multimodal deep reinforcement learning model for inferring pathological features and the diagnosis of AD. Approach. First, for better clinical feasibility, the compressed-sensing MRI image is reconstructed using an interpretable deep reinforcement learning model. Then, the reconstructed MRI is input into the full convolution neural network to generate a pixel-level disease probability risk map (DPM) of the whole brain for AD. The DPM of important brain regions and individual information are then input into the attention-based fully deep neural network to obtain the diagnosis results and analyze the biomarkers. We used 1349 multi-center samples to construct and test the model. Main results. Finally, the model obtained 99.6% ± 0.2%, 97.9% ± 0.2%, and 96.1% ± 0.3% area under curve in ADNI, AIBL and NACC, respectively. The model also provides an effective analysis of multimodal pathology, predicts the imaging biomarkers in MRI and the weight of each individual item of information. In this study, a deep reinforcement learning model was designed, which can not only accurately diagnose AD, but analyze potential biomarkers. Significance. In this study, a deep reinforcement learning model was designed. The model builds a bridge between clinical practice and AI diagnosis and provides a viewpoint for the interpretability of AI technology.

Prediction of central venous catheter-associated deep venous thrombosis in pediatric critical care settings

BackgroundAn increase in the incidence of central venous catheter (CVC)-associated deep venous thrombosis (CADVT) has been reported in pediatric patients over the past decade. At the same time, current screening guidelines for venous thromboembolism risk have low sensitivity for CADVT in hospitalized children. This study utilized a multimodal deep learning model to predict CADVT before it occurs.MethodsChildren who were admitted to intensive care units (ICUs) between December 2015 and December 2018 and with CVC placement at least 3 days were included. The variables analyzed included demographic characteristics, clinical conditions, laboratory test results, vital signs and medications. A multimodal deep learning (MMDL) model that can handle temporal data using long short-term memory (LSTM) and gated recurrent units (GRUs) was proposed for this prediction task. Four benchmark machine learning models, logistic regression (LR), random forest (RF), gradient boosting decision tree (GBDT) and a published cutting edge MMDL, were used to compare and evaluate the models with a fivefold cross-validation approach. Accuracy, recall, area under the ROC curve (AUC), and average precision (AP) were used to evaluate the discrimination of each model at three time points (24 h, 48 h and 72 h) before CADVT occurred. Brier score and Spiegelhalter’s z test were used measure the calibration of these prediction models.ResultsA total of 1830 patients were included in this study, and approximately 15% developed CADVT. In the CADVT prediction task, the model proposed in this paper significantly outperforms both traditional machine learning models and existing multimodal deep learning models at all 3 time points. It achieved 77% accuracy and 90% recall at 24 h before CADVT was discovered. It can be used to accurately predict the occurrence of CADVT 72 h in advance with an accuracy of greater than 75%, a recall of more than 87%, and an AUC value of 0.82.ConclusionIn this study, a machine learning method was successfully established to predict CADVT in advance. These findings demonstrate that artificial intelligence (AI) could provide measures for thromboprophylaxis in a pediatric intensive care setting.

Multimodal Deep Learning for Prognosis Prediction in Renal Cancer.

BackgroundClear-cell renal cell carcinoma (ccRCC) is common and associated with substantial mortality. TNM stage and histopathological grading have been the sole determinants of a patient’s prognosis for decades and there are few prognostic biomarkers used in clinical routine. Management of ccRCC involves multiple disciplines such as urology, radiology, oncology, and pathology and each of these specialties generates highly complex medical data. Here, artificial intelligence (AI) could prove extremely powerful to extract meaningful information to benefit patients.ObjectiveIn the study, we developed and evaluated a multimodal deep learning model (MMDLM) for prognosis prediction in ccRCC.Design, Setting, and ParticipantsTwo mixed cohorts of non-metastatic and metastatic ccRCC patients were used: (1) The Cancer Genome Atlas cohort including 230 patients and (2) the Mainz cohort including 18 patients with ccRCC. For each of these patients, we trained the MMDLM on multiscale histopathological images, CT/MRI scans, and genomic data from whole exome sequencing.Outcome Measurements and Statistical AnalysisOutcome measurements included Harrell’s concordance index (C-index) and also various performance parameters for predicting the 5-year survival status (5YSS). Different visualization techniques were used to make our model more transparent.ResultsThe MMDLM showed great performance in predicting the prognosis of ccRCC patients with a mean C-index of 0.7791 and a mean accuracy of 83.43%. Training on a combination of data from different sources yielded significantly better results compared to when only one source was used. Furthermore, the MMDLM’s prediction was an independent prognostic factor outperforming other clinical parameters.InterpretationMultimodal deep learning can contribute to prognosis prediction in ccRCC and potentially help to improve the clinical management of this disease.Patient SummaryAn AI-based computer program can analyze various medical data (microscopic images, CT/MRI scans, and genomic data) simultaneously and thereby predict the survival time of patients with renal cancer.