Area Under The Precision-recall Curve Research Articles

Utilizing imaging parameters for functional outcome prediction in acute ischemic stroke: A machine learning study.

We aimed to predict the functional outcome of acute ischemic stroke patients with anterior circulation large vessel occlusions (LVOs), irrespective of how they were treated or the severity of the stroke at admission, by only using imaging parameters in machine learning models. Consecutive adult patients with anterior circulation LVOs who were scanned with CT angiography (CTA) and CT perfusion were queried in this single-center, retrospective study. The favorable outcome was defined as a modified Rankin score (mRS) of 0-2 at 90 days. Predictor variables included only imaging parameters. CatBoost, XGBoost, and Random Forest were employed. Algorithms were evaluated using the area under the receiver operating characteristic curve (AUROC), the area under the precision-recall curve (AUPRC), accuracy, Brier score, recall, and precision. SHapley Additive exPlanations were implemented. A total of 180 patients (102 female) were included, with a median age of 69.5. Ninety-two patients had an mRS between 0 and 2. The best algorithm in terms of AUROC was XGBoost (0.91). Furthermore, the XGBoost model exhibited a precision of 0.72, a recall of 0.81, an AUPRC of 0.83, an accuracy of 0.78, and a Brier score of 0.17. Multiphase CTA collateral score was the most significant feature in predicting the outcome. Using only imaging parameters, our model had an AUROC of 0.91 which was superior to most previous studies, indicating that imaging parameters may be as accurate as conventional predictors. The multiphase CTA collateral score was the most predictive variable, highlighting the importance of collaterals.

Automated Machine Learning versus Expert-Designed Models in Ocular Toxoplasmosis: Detection and Lesion Localization Using Fundus Images

ABSTRACT Purpose Automated machine learning (AutoML) allows clinicians without coding experience to build their own deep learning (DL) models. This study assesses the performance of AutoML in detecting and localizing ocular toxoplasmosis (OT) lesions in fundus images and compares it to expert-designed models. Methods Ophthalmology trainees without coding experience designed AutoML models using 304 labelled fundus images. We designed a binary model to differentiate OT from normal and an object detection model to visually identify OT lesions. Results The AutoML model had an area under the precision-recall curve (AuPRC) of 0.945, sensitivity of 100%, specificity of 83% and accuracy of 93.5% (vs. 94%, 86% and 91% for the bespoke models). The AutoML object detection model had an AuPRC of 0.600 with a precision of 93.3% and recall of 56%. Using a diversified external validation dataset, our model correctly labeled 15 normal fundus images (100%) and 15 OT fundus images (100%), with a mean confidence score of 0.965 and 0.963, respectively. Conclusion AutoML models created by ophthalmologists without coding experience were comparable or better than expert-designed bespoke models trained on the same dataset. By creatively using AutoML to identify OT lesions on fundus images, our approach brings the whole spectrum of DL model design into the hands of clinicians.

A User-friendly Approach for the Diagnosis of Diabetic Retinopathy Using ChatGPT and Automated Machine Learning

To assess the capabilities of Chat Generative Pre-trained Transformer (ChatGPT) and Vertex AI in executing code-free preprocessing, training machine learning (ML) models, and analyzing the data. Evaluation of diagnostic test or technology. ChatGPT and Vetrex AI as publicly available large language model and ML platform, respectively. ChatGPT was employed to improve the resolution of fundus photography images from the Methods to Evaluate Segmentation and Indexing Techniques in the field of Retinal Ophthalmology (Messidor-2) open-source dataset using the Contrast Limited Adaptive Histogram Equalization (CLAHE) technique by Fiji software. Subsequently, Vertex AI, an automated ML (AutoML) platform, was utilized to develop 2 classification models. The first model served as a binary classifier for detecting the presence of diabetic retinopathy (DR), while the second determined its severity. Finally, ChatGPT was used to provide scripts for R and Python programming languages for data analysis and was also directly employed in analyzing the data in a code-free method. Evaluating the utility of ChatGPT in generating scripts for preprocessing images using Fiji and analyzing data across Python and R and assessing its potential in analyzing data through a code-free method. Investigating the capabilities of Vertex AI to train image classification models for detection of DR and its severity. Two ML models were trained using 1740 images from the Messidor-2 database. The first model, designed to detect the severity of DR, achieved an area under the precision-recall curve (AUPRC) of 0.81, with a precision rate of 81.81% and recall of 72.83%. The second model, tailored for the detection of the presence of DR, recorded a precision and recall of 84.48% with an AUPRC of 0.90. ChatGPT and Vertex AI have the potential to enable physicians without coding expertise to preprocess images, analyze data, and train ML models. Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Classification of self-limited epilepsy with centrotemporal spikes by classical machine learning and deep learning based on electroencephalogram data

Electroencephalogram (EEG) has been widely utilized as a valuable assessment tool for diagnosing epilepsy in hospital settings. However, clinical diagnosis of patients with self-limited epilepsy with centrotemporal spikes (SeLECTS) is challenging due to the presence of similar abnormal discharges in EEG displays compared to other types of epilepsy (non-SeLECTS) patients. To assist the diagnostic process of epilepsy, a comprehensive classification study utilizing machine learning or deep learning techniques is proposed. In this study, clinical EEG was collected from 33 patients diagnosed with either SeLECTS or non-SeLECTS, aged between 3 and 11 years. In the realm of classical machine learning, sharp wave features (including upslope, downslope, and width at half maximum) were extracted from the EEG data. These features were then combined with the random forest (RF) and extreme random forest (ERF) classifiers to differentiate between SeLECTS and non-SeLECTS. Additionally, deep learning was employed by directly inputting the EEG data into a deep residual network (ResNet) for classification. The classification results were evaluated based on accuracy, F1-score, area under the curve (AUC), and area under the precision-recall curve (AUPRC). Following a 10-fold cross-validation, the ERF classifier achieved an accuracy of 73.15 % when utilizing sharp wave feature extraction for classification. The F1-score obtained was 0.72, while the AUC and AUPRC values were 0.75 and 0.63, respectively. On the other hand, the ResNet model achieved a classification accuracy of 90.49 %, with an F1-score of 0.90. The AUC and AUPRC values for ResNet were found to be 0.96 and 0.92, respectively. These results highlighted the significant potential of deep learning methods in SeLECTS classification research, owing to their high accuracy. Moreover, feature extraction-based methods demonstrated good reliability and could assist in identifying relevant biological features of SeLECTS within EEG data.

Automated Machine Learning for Predicting Diabetic Retinopathy Progression From Ultra-Widefield Retinal Images

Machine learning (ML) algorithms have the potential to identify eyes with early diabetic retinopathy (DR) at increased risk for disease progression. To create and validate automated ML models (autoML) for DR progression from ultra-widefield (UWF) retinal images. Deidentified UWF images with mild or moderate nonproliferative DR (NPDR) with 3 years of longitudinal follow-up retinal imaging or evidence of progression within 3 years were used to develop automated ML models for predicting DR progression in UWF images. All images were collected from a tertiary diabetes-specific medical center retinal image dataset. Data were collected from July to September 2022. Automated ML models were generated from baseline on-axis 200° UWF retinal images. Baseline retinal images were labeled for progression based on centralized reading center evaluation of baseline and follow-up images according to the clinical Early Treatment Diabetic Retinopathy Study severity scale. Images for model development were split 8-1-1 for training, optimization, and testing to detect 1 or more steps of DR progression. Validation was performed using a 328-image set from the same patient population not used in model development. Area under the precision-recall curve (AUPRC), sensitivity, specificity, and accuracy. A total of 1179 deidentified UWF images with mild (380 [32.2%]) or moderate (799 [67.8%]) NPDR were included. DR progression was present in half of the training set (590 of 1179 [50.0%]). The model's AUPRC was 0.717 for baseline mild NPDR and 0.863 for moderate NPDR. On the validation set for eyes with mild NPDR, sensitivity was 0.72 (95% CI, 0.57-0.83), specificity was 0.63 (95% CI, 0.57-0.69), prevalence was 0.15 (95% CI, 0.12-0.20), and accuracy was 64.3%; for eyes with moderate NPDR, sensitivity was 0.80 (95% CI, 0.70-0.87), specificity was 0.72 (95% CI, 0.66-0.76), prevalence was 0.22 (95% CI, 0.19-0.27), and accuracy was 73.8%. In the validation set, 6 of 9 eyes (75%) with mild NPDR and 35 of 41 eyes (85%) with moderate NPDR progressed 2 steps or more were identified. All 4 eyes with mild NPDR that progressed within 6 months and 1 year were identified, and 8 of 9 (89%) and 17 of 20 (85%) with moderate NPDR that progressed within 6 months and 1 year, respectively, were identified. This study demonstrates the accuracy and feasibility of automated ML models for identifying DR progression developed using UWF images, especially for prediction of 2-step or greater DR progression within 1 year. Potentially, the use of ML algorithms may refine the risk of disease progression and identify those at highest short-term risk, thus reducing costs and improving vision-related outcomes.

Pediatric ECG-Based Deep Learning to Predict Left Ventricular Dysfunction and Remodeling.

Artificial intelligence-enhanced ECG analysis shows promise to detect ventricular dysfunction and remodeling in adult populations. However, its application to pediatric populations remains underexplored. A convolutional neural network was trained on paired ECG-echocardiograms (≤2 days apart) from patients ≤18 years of age without major congenital heart disease to detect human expert-classified greater than mild left ventricular (LV) dysfunction, hypertrophy, and dilation (individually and as a composite outcome). Model performance was evaluated on single ECG-echocardiogram pairs per patient at Boston Children's Hospital and externally at Mount Sinai Hospital using area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC). The training cohort comprised 92 377 ECG-echocardiogram pairs (46 261 patients; median age, 8.2 years). Test groups included internal testing (12 631 patients; median age, 8.8 years; 4.6% composite outcomes), emergency department (2830 patients; median age, 7.7 years; 10.0% composite outcomes), and external validation (5088 patients; median age, 4.3 years; 6.1% composite outcomes) cohorts. Model performance was similar on internal test and emergency department cohorts, with model predictions of LV hypertrophy outperforming the pediatric cardiologist expert benchmark. Adding age and sex to the model added no benefit to model performance. When using quantitative outcome cutoffs, model performance was similar between internal testing (composite outcome: AUROC, 0.88, AUPRC, 0.43; LV dysfunction: AUROC, 0.92, AUPRC, 0.23; LV hypertrophy: AUROC, 0.88, AUPRC, 0.28; LV dilation: AUROC, 0.91, AUPRC, 0.47) and external validation (composite outcome: AUROC, 0.86, AUPRC, 0.39; LV dysfunction: AUROC, 0.94, AUPRC, 0.32; LV hypertrophy: AUROC, 0.84, AUPRC, 0.25; LV dilation: AUROC, 0.87, AUPRC, 0.33), with composite outcome negative predictive values of 99.0% and 99.2%, respectively. Saliency mapping highlighted ECG components that influenced model predictions (precordial QRS complexes for all outcomes; T waves for LV dysfunction). High-risk ECG features include lateral T-wave inversion (LV dysfunction), deep S waves in V1 and V2 and tall R waves in V6 (LV hypertrophy), and tall R waves in V4 through V6 (LV dilation). This externally validated algorithm shows promise to inexpensively screen for LV dysfunction and remodeling in children, which may facilitate improved access to care by democratizing the expertise of pediatric cardiologists.

A subregion-based RadioFusionOmics model discriminates between grade 4 astrocytoma and glioblastoma on multisequence MRI

PurposeTo explore a subregion-based RadioFusionOmics (RFO) model for discrimination between adult-type grade 4 astrocytoma and glioblastoma according to the 2021 WHO CNS5 classification.Methods329 patients (40 grade 4 astrocytomas and 289 glioblastomas) with histologic diagnosis was retrospectively collected from our local institution and The Cancer Imaging Archive (TCIA). The volumes of interests (VOIs) were obtained from four multiparametric MRI sequences (T1WI, T1WI + C, T2WI, T2-FLAIR) using (1) manual segmentation of the non-enhanced tumor (nET), enhanced tumor (ET), and peritumoral edema (pTE), and (2) K-means clustering of four habitats (H1: high T1WI + C, high T2-FLAIR; (2) H2: high T1WI + C, low T2-FLAIR; (3) H3: low T1WI + C, high T2-FLAIR; and (4) H4: low T1WI + C, low T2-FLAIR). The optimal VOI and best MRI sequence combination were determined. The performance of the RFO model was evaluated using the area under the precision-recall curve (AUPRC) and the best signatures were identified.ResultsThe two best VOIs were manual VOI3 (putative peritumoral edema) and clustering H34 (low T1WI + C, high T2-FLAIR (H3) combined with low T1WI + C and low T2-FLAIR (H4)). Features fused from four MRI sequences (Fseq1,2,3,4\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${F}_{seq}^{\\mathrm{1,2},\\mathrm{3,4}}$$\\end{document}) outperformed those from either a single sequence or other sequence combinations. The RFO model that was trained using fused features Fseq1,2,3,4\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${F}_{seq}^{\\mathrm{1,2},\\mathrm{3,4}}$$\\end{document} achieved the AUPRC of 0.972 (VOI3) and 0.976 (H34) in the primary cohort (p = 0.905), and 0.971 (VOI3) and 0.974 (H34) in the testing cohort (p = 0.402).ConclusionThe performance of subregions defined by clustering was comparable to that of subregions that were manually defined. Fusion of features from the edematous subregions of multiple MRI sequences by the RFO model resulted in differentiation between grade 4 astrocytoma and glioblastoma.

A 10-year retrospective cohort of diabetic patients in a large medical institution: Utilizing multiple machine learning models for diabetic kidney disease prediction.

As the prevalence of diabetes steadily increases, the burden of diabetic kidney disease (DKD) is also intensifying. In response, we have utilized a 10-year diabetes cohort from our medical center to train machine learning-based models for predicting DKD and interpreting relevant factors. Employing a large dataset from 73,101 hospitalized type 2 diabetes patients at The First Affiliated Hospital of Zhengzhou University, we analyzed demographic and medication data. Machine learning models, including XGBoost, CatBoost, LightGBM, Random Forest, AdaBoost, GBDT (gradient boosting decision tree), and SGD (stochastic gradient descent), were trained on these data, focusing on interpretability by SHAP. SHAP explains the output of the models by assigning an importance value to each feature for a particular prediction, enabling a clear understanding of how individual features influence the prediction outcomes. The XGBoost model achieved an area under the curve (AUC) of 0.95 and an area under the precision-recall curve (AUPR) of 0.76, while CatBoost recorded an AUC of 0.97 and an AUPR of 0.84. These results underscore the effectiveness of these models in predicting DKD in patients with type 2 diabetes. This study provides a comprehensive approach for predicting DKD in patients with type 2 diabetes, employing machine learning techniques. The findings are crucial for the early detection and intervention of DKD, offering a roadmap for future research and healthcare strategies in diabetes management. Additionally, the presence of non-diabetic kidney diseases and diabetes with complications was identified as significant factors in the development of DKD.

Identifying the need for infection-related consultations in intensive care patients using machine learning models

Infection-related consultations on intensive care units (ICU) have a positive impact on quality of care and clinical outcome. However, timing of these consultations is essential and to date they are typically event-triggered and reactive. Here, we investigate a proactive approach to identify patients in need for infection-related consultations by machine learning models using routine electronic health records. Data was retrieved from a mixed ICU at a large academic tertiary care hospital including 9684 admissions. Infection-related consultations were predicted using logistic regression, random forest, gradient boosting machines, and long short-term memory neural networks (LSTM). Overall, 7.8% of admitted patients received an infection-related consultation. Time-sensitive modelling approaches performed better than static approaches. Using LSTM resulted in the prediction of infection-related consultations in the next clinical shift (up to eight hours in advance) with an area under the receiver operating curve (AUROC) of 0.921 and an area under the precision recall curve (AUPRC) of 0.541. The successful prediction of infection-related consultations for ICU patients was done without the use of classical triggers, such as (interim) microbiology reports. Predicting this key event can potentially streamline ICU and consultant workflows and improve care as well as outcome for critically ill patients with (suspected) infections.

Development and validation of machine learning-augmented algorithm for insulin sensitivity assessment in the community and primary care settings: a population-based study in China.

Insulin plays a central role in the regulation of energy and glucose homeostasis, and insulin resistance (IR) is widely considered as the "common soil" of a cluster of cardiometabolic disorders. Assessment of insulin sensitivity is very important in preventing and treating IR-related disease. This study aims to develop and validate machine learning (ML)-augmented algorithms for insulin sensitivity assessment in the community and primary care settings. We analyzed the data of 9358 participants over 40 years old who participated in the population-based cohort of the Hubei center of the REACTION study (Risk Evaluation of Cancers in Chinese Diabetic Individuals). Three non-ensemble algorithms and four ensemble algorithms were used to develop the models with 70 non-laboratory variables for the community and 87 (70 non-laboratory and 17 laboratory) variables for the primary care settings to screen the classifier of the state-of-the-art. The models with the best performance were further streamlined using top-ranked 5, 8, 10, 13, 15, and 20 features. Performances of these ML models were evaluated using the area under the receiver operating characteristic curve (AUROC), the area under the precision-recall curve (AUPR), and the Brier score. The Shapley additive explanation (SHAP) analysis was employed to evaluate the importance of features and interpret the models. The LightGBM models developed for the community (AUROC 0.794, AUPR 0.575, Brier score 0.145) and primary care settings (AUROC 0.867, AUPR 0.705, Brier score 0.119) achieved higher performance than the models constructed by the other six algorithms. The streamlined LightGBM models for the community (AUROC 0.791, AUPR 0.563, Brier score 0.146) and primary care settings (AUROC 0.863, AUPR 0.692, Brier score 0.124) using the 20 top-ranked variables also showed excellent performance. SHAP analysis indicated that the top-ranked features included fasting plasma glucose (FPG), waist circumference (WC), body mass index (BMI), triglycerides (TG), gender, waist-to-height ratio (WHtR), the number of daughters born, resting pulse rate (RPR), etc. The ML models using the LightGBM algorithm are efficient to predict insulin sensitivity in the community and primary care settings accurately and might potentially become an efficient and practical tool for insulin sensitivity assessment in these settings.

Time-Series Aware Metrics for the Evaluation of Intraoperative Electroencephalography-Based Ischemia Detection.

Continuous intraoperative monitoring with electroencephalo2 graphy (EEG) is commonly used to detect cerebral ischemia in high-risk surgical procedures such as carotid endarterectomy. Machine learning (ML) models that detect ischemia in real time can form the basis of automated intraoperative EEG monitoring. In this study, we describe and compare two time-series aware precision and recall metrics to the classical precision and recall metrics for evaluating the performance of ML models that detect ischemia. We trained six ML models to detect ischemia in intraoperative EEG and evaluated them with the area under the precision-recall curve (AUPRC) using time-series aware and classical approaches to compute precision and recall. The Support Vector Classification (SVC) model performed the best on the time-series aware metrics, while the Light Gradient Boosting Machine (LGBM) model performed the best on the classical metrics. Visual inspection of the probability outputs of the models alongside the actual ischemic periods revealed that the time-series aware AUPRC selected a model more likely to predict ischemia onset in a timely fashion than the model selected by classical AUPRC.

GMFGRN: a matrix factorization and graph neural network approach for gene regulatory network inference.

The recent advances of single-cell RNA sequencing (scRNA-seq) have enabled reliable profiling of gene expression at the single-cell level, providing opportunities for accurate inference of gene regulatory networks (GRNs) on scRNA-seq data. Most methods for inferring GRNs suffer from the inability to eliminate transitive interactions or necessitate expensive computational resources. To address these, we present a novel method, termed GMFGRN, for accurate graph neural network (GNN)-based GRN inference from scRNA-seq data. GMFGRN employs GNN for matrix factorization and learns representative embeddings for genes. For transcription factor-gene pairs, it utilizes the learned embeddings to determine whether they interact with each other. The extensive suite of benchmarking experiments encompassing eight static scRNA-seq datasets alongside several state-of-the-art methods demonstrated mean improvements of 1.9 and 2.5% over the runner-up in area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC). In addition, across four time-series datasets, maximum enhancements of 2.4 and 1.3% in AUROC and AUPRC were observed in comparison to the runner-up. Moreover, GMFGRN requires significantly less training time and memory consumption, with time and memory consumed <10% compared to the second-best method. These findings underscore the substantial potential of GMFGRN in the inference of GRNs. It is publicly available at https://github.com/Lishuoyy/GMFGRN.

KGE-UNIT: toward the unification of molecular interactions prediction based on knowledge graph and multi-task learning on drug discovery.

The prediction of molecular interactions is vital for drug discovery. Existing methods often focus on individual prediction tasks and overlook the relationships between them. Additionally, certain tasks encounter limitations due to insufficient data availability, resulting in limited performance. To overcome these limitations, we propose KGE-UNIT, a unified framework that combines knowledge graph embedding (KGE) and multi-task learning, for simultaneous prediction of drug-target interactions (DTIs) and drug-drug interactions (DDIs) and enhancing the performance of each task, even when data availability is limited. Via KGE, we extract heterogeneous features from the drug knowledge graph to enhance the structural features of drug and protein nodes, thereby improving the quality of features. Additionally, employing multi-task learning, we introduce an innovative predictor that comprises the task-aware Convolutional Neural Network-based (CNN-based) encoder and the task-aware attention decoder which can fuse better multimodal features, capture the contextual interactions of molecular tasks and enhance task awareness, leading to improved performance. Experiments on two imbalanced datasets for DTIs and DDIs demonstrate the superiority of KGE-UNIT, achieving high area under the receiver operating characteristics curves (AUROCs) (0.942, 0.987) and area under the precision-recall curve ( AUPRs) (0.930, 0.980) for DTIs and high AUROCs (0.975, 0.989) and AUPRs (0.966, 0.988) for DDIs. Notably, on the LUO dataset where the data were more limited, KGE-UNIT exhibited a more pronounced improvement, with increases of 4.32$\%$ in AUROC and 3.56$\%$ in AUPR for DTIs and 6.56$\%$ in AUROC and 8.17$\%$ in AUPR for DDIs. The scalability of KGE-UNIT is demonstrated through its extension to protein-protein interactions prediction, ablation studies and case studies further validate its effectiveness.

Predicting Drug-drug Interaction with Graph Mutual Interaction Attention Mechanism

Effective representation of molecules is a crucial step in AI-driven drug design and drug discovery, especially for drug-drug interaction (DDIs) prediction. Previous work usually models the drug information from the drug-related knowledge graph or the single drug molecules, but the interaction information between molecular substructures of drug pair is seldom considered, thus often ignoring the influence of bond information on atom node representation, leading to insufficient drug representation. Moreover, key molecular substructures have significant contribution to the DDIs prediction results. Therefore, in this work, we propose a novel Graph learning framework of Mutual Interaction Attention mechanism (called GMIA) to predict DDIs by effectively representing the drug molecules. Specifically, we build the node-edge message communication encoder to aggregate atom node and the incoming edge information for atom node representation and design the mutual interaction attention decoder to capture the mutual interaction context between molecular graphs of drug pairs. GMIA can bridge the gap between two encoders for the single drug molecules by attention mechanism. We also design a co-attention matrix to analyze the significance of different-size substructures obtained from the encoder-decoder layer and provide interpretability. In comparison with other recent state-of-the-art methods, our GMIA achieves the best results in terms of area under the precision-recall-curve (AUPR), area under the ROC curve (AUC), and F1 score on two different scale datasets. The case study indicates that our GMIA can detect the key substructure for potential DDIs, demonstrating the enhanced performance and interpretation ability of GMIA.

Early prediction of pediatric asthma in the Canadian Healthy Infant Longitudinal Development (CHILD) birth cohort using machine learning

BackgroundEarly identification of children at risk of asthma can have significant clinical implications for effective intervention and treatment. This study aims to disentangle the relative timing and importance of early markers of asthma.MethodsUsing the CHILD Cohort Study, 132 variables measured in 1754 multi-ethnic children were included in the analysis for asthma prediction. Data up to 4 years of age was used in multiple machine learning models to predict physician-diagnosed asthma at age 5 years. Both predictive performance and variable importance was assessed in these models.ResultsEarly-life data (≤1 year) has limited predictive ability for physician-diagnosed asthma at age 5 years (area under the precision-recall curve (AUPRC) < 0.35). The earliest reliable prediction of asthma is achieved at age 3 years, (area under the receiver-operator curve (AUROC) > 0.90) and (AUPRC > 0.80). Maternal asthma, antibiotic exposure, and lower respiratory tract infections remained highly predictive throughout childhood. Wheezing status and atopy are the most important predictors of early childhood asthma from among the factors included in this study.ConclusionsChildhood asthma is predictable from non-biological measurements from the age of 3 years, primarily using parental asthma and patient history of wheezing, atopy, antibiotic exposure, and lower respiratory tract infections.ImpactMachine learning models can predict physician-diagnosed asthma in early childhood (AUROC > 0.90 and AUPRC > 0.80) using ≥3 years of non-biological and non-genetic information, whereas prediction with the same patient information available before 1 year of age is challenging.Wheezing, atopy, antibiotic exposure, lower respiratory tract infections, and the child’s mother having asthma were the strongest early markers of 5-year asthma diagnosis, suggesting an opportunity for earlier diagnosis and intervention and focused assessment of patients at risk for asthma, with an evolving risk stratification over time.

An interpretable machine learning model for stroke recurrence in patients with symptomatic intracranial atherosclerotic arterial stenosis.

Symptomatic intracranial atherosclerotic stenosis (SICAS) is the most common etiology of ischemic stroke and one of the main causes of high stroke recurrence. The recurrence of stroke is closely related to the prognosis of ischemic stroke. This study aims to develop a machine learning model based on high-resolution vessel wall imaging (HR-VWI) to predict the risk of stroke recurrence in SICAS. This study retrospectively collected data from 180 SICAS stroke patients treated at the hospital between 2020.01 and 2022.01. Relevant imaging and clinical data were collected, and follow-up was conducted. The dataset was divided into a training set and a validation set in a ratio of 7:3. We employed the least absolute shrinkage and selection operator (LASSO) regression to perform a selection on the baseline data, laboratory tests, and neuroimaging data generated by HR-VWI scans collected from the training set. Finally, five machine learning techniques, including logistic regression model (LR), support vector machine (SVM), Gaussian naive Bayes (GNB), Complement naive Bayes (CNB), and k-nearest neighbors algorithm (kNN), were employed to develop a predictive model for stroke recurrence. Shapley Additive Explanation (SHAP) was used to provide visualization and interpretation for each patient. The model's effectiveness was evaluated using average accuracy, sensitivity, specificity, precision, f1 score, PR curve, calibration curve, and decision curve analysis. LASSO analysis revealed that "history of hypertension," "homocysteine level," "NWI value," "stenosis rate," "intracranial hemorrhage," "positive remodeling," and "enhancement grade" were independent risk factors for stroke recurrence in SICAS patients. In 10-fold cross-validation, the area under the curve (AUC) ranged from 0.813 to 0.912 in ROC curve analysis. The area under the precision-recall curve (AUPRC) ranged from 0.655 to 0.833, with the Gaussian Naive Bayes (GNB) model exhibiting the best ability to predict stroke recurrence in SICAS. SHAP analysis provided interpretability for the machine learning model and revealed essential factors related to the risk of stroke recurrence in SICAS. A precise machine learning-based prediction model for stroke recurrence in SICAS has been established to assist clinical practitioners in making clinical decisions and implementing personalized treatment measures.

InterDILI: interpretable prediction of drug-induced liver injury through permutation feature importance and attention mechanism

Safety is one of the important factors constraining the distribution of clinical drugs on the market. Drug-induced liver injury (DILI) is the leading cause of safety problems produced by drug side effects. Therefore, the DILI risk of approved drugs and potential drug candidates should be assessed. Currently, in vivo and in vitro methods are used to test DILI risk, but both methods are labor-intensive, time-consuming, and expensive. To overcome these problems, many in silico methods for DILI prediction have been suggested. Previous studies have shown that DILI prediction models can be utilized as prescreening tools, and they achieved a good performance. However, there are still limitations in interpreting the prediction results. Therefore, this study focused on interpreting the model prediction to analyze which features could potentially cause DILI. For this, five publicly available datasets were collected to train and test the model. Then, various machine learning methods were applied using substructure and physicochemical descriptors as inputs and the DILI label as the output. The interpretation of feature importance was analyzed by recognizing the following general-to-specific patterns: (i) identifying general important features of the overall DILI predictions, and (ii) highlighting specific molecular substructures which were highly related to the DILI prediction for each compound. The results indicated that the model not only captured the previously known properties to be related to DILI but also proposed a new DILI potential substructural of physicochemical properties. The models for the DILI prediction achieved an area under the receiver operating characteristic (AUROC) of 0.88–0.97 and an area under the Precision-Recall curve (AUPRC) of 0.81–0.95. From this, we hope the proposed models can help identify the potential DILI risk of drug candidates at an early stage and offer valuable insights for drug development.

Interpretable machine learning model for early prediction of 28-day mortality in ICU patients with sepsis-induced coagulopathy: development and validation

ObjectiveSepsis-induced coagulopathy (SIC) is extremely common in individuals with sepsis, significantly associated with poor outcomes. This study attempted to develop an interpretable and generalizable machine learning (ML) model for early predicting the risk of 28-day death in patients with SIC.MethodsIn this retrospective cohort study, we extracted SIC patients from the Medical Information Mart for Intensive Care III (MIMIC-III), MIMIC-IV, and eICU-CRD database according to Toshiaki Iba's scale. And the overlapping in the MIMIC-IV was excluded for this study. Afterward, only the MIMIC-III cohort was randomly divided into the training set, and the internal validation set according to the ratio of 7:3, while the MIMIC-IV and eICU-CRD databases were considered the external validation sets. The predictive factors for 28-day mortality of SIC patients were determined using recursive feature elimination combined with tenfold cross-validation (RFECV). Then, we constructed models using ML algorithms. Multiple metrics were used for evaluation of performance of the models, including the area under the receiver operating characteristic curve (AUROC), area under the precision recall curve (AUPRC), accuracy, sensitivity, specificity, negative predictive value, positive predictive value, recall, and F1 score. Finally, Shapley Additive Explanations (SHAP), Local Interpretable Model-Agnostic Explanations (LIME) were employed to provide a reasonable interpretation for the prediction results.ResultsA total of 3280, 2798, and 1668 SIC patients were screened from MIMIC-III, MIMIC-IV, and eICU-CRD databases, respectively. Seventeen features were selected to construct ML prediction models. XGBoost had the best performance in predicting the 28-day mortality of SIC patients, with AUC of 0.828, 0.913 and 0.923, the AUPRC of 0.807, 0.796 and 0.921, the accuracy of 0.785, 0.885 and 0.891, the F1 scores were 0.63, 0.69 and 0.70 in MIMIC-III (internal validation set), MIMIC-IV, and eICU-CRD databases. The importance ranking and SHAP analyses showed that initial SOFA score, red blood cell distribution width (RDW), and age were the top three critical features in the XGBoost model.ConclusionsWe developed an optimal and explainable ML model to predict the risk of 28-day death of SIC patients 28-day death risk. Compared with conventional scoring systems, the XGBoost model performed better. The model established will have the potential to improve the level of clinical practice for SIC patients.Graphical

Machine learning approaches for predicting frailty base on multimorbidities in US adults using NHANES data (1999–2018)

BackgroundThe global increase in an aging population has led to more common age-related health challenges, particularly multimorbidity and frailty, but there is a significant gap. MethodsThis cross-sectional study utilized data from the National Health and Nutrition Examination Survey (1999–2018). The association between age and frailty was assessed using a restricted cubic spline (RCS) model, while weighted adjusted multivariable logistic regression evaluated the effect of diseases to frailty. And in machine learning process, feature selection for the frailty prediction model involved three algorithms. The model's performance was optimized using nested cross-validation and tested with various algorithms including decision tree, Logistic Regression, k-Nearest Neighbor, Random Forest, Recursive Partitioning and Regression Trees, and eXtreme Gradient Boosting (XGBoost). We used areas under the receiver operating characteristic curve (AUC) and area under the precision-recall curve (AU-PRC) to evaluate six algorithms, select the optimal model, and test the discrimination and consistency of the optimal model. ResultsThe study included 46,187 participants, with 6,009 cases of frailty. RCS analysis showed a non-linear association between age and frailty, with a turning point at 49 years. Key impacting variables identified are Anemia, Arthritis, Diabetes Mellitus, Coronary Heart Disease, and Hypertension. In the machine learning process, we selected the optimal data set by feature selection, including 13 variables. Through nested cross-validation, a total of 31,900 models were built using 6 algorithms. And the XGBoost model showed the highest performance (AUC = 0.8828 and AU-PRC = 0.624), and clear proficiency in both discrimination and calibration. ConclusionsWe found 49 years maintain the balance of physiological reserve and external aggression. In addition, chronic diseases are trigger factor of frailty, while acute diseases are contributing factor that exacerbates the body's rapid decline. Last, the XGBoost frailty prediction model, with its simplicity, high performance and high clinical value holds potential for clinical application.

External evaluation of the Dynamic Criticality Index: A machine learning model to predict future need for ICU care in hospitalized pediatric patients.

To assess the single site performance of the Dynamic Criticality Index (CI-D) models developed from a multi-institutional database to predict future care. Secondarily, to assess future care-location predictions in a single institution when CI-D models are re-developed using single-site data with identical variables and modeling methods. Four CI-D models were assessed for predicting care locations >6-12 hours, >12-18 hours, >18-24 hours, and >24-30 hours in the future. Prognostic study comparing multi-institutional CI-D models' performance in a single-site electronic health record dataset to an institution-specific CI-D model developed using identical variables and modelling methods. The institution did not participate in the multi-institutional dataset. All pediatric inpatients admitted from January 1st 2018 -February 29th 2020 through the emergency department. The main outcome was inpatient care in routine or ICU care locations. A total of 29,037 pediatric hospital admissions were included, with 5,563 (19.2%) admitted directly to the ICU, 869 (3.0%) transferred from routine to ICU care, and 5,023 (17.3%) transferred from ICU to routine care. Patients had a median [IQR] age 68 months (15-157), 47.5% were female and 43.4% were black. The area under the receiver operating characteristic curve (AUROC) for the multi-institutional CI-D models applied to a single-site test dataset was 0.493-0.545 and area under the precision-recall curve (AUPRC) was 0.262-0.299. The single-site CI-D models applied to an independent single-site test dataset had an AUROC 0.906-0.944 and AUPRC range from 0.754-0.824. Accuracy at 0.95 sensitivity for those transferred from routine to ICU care was 72.6%-81.0%. Accuracy at 0.95 specificity was 58.2%-76.4% for patients who transferred from ICU to routine care. Models developed from multi-institutional datasets and intended for application to individual institutions should be assessed locally and may benefit from re-development with site-specific data prior to deployment.