Area Under The Precision-recall Curve Research Articles

Interpretable machine learning models for hospital readmission prediction: a two-step extracted regression tree approach

BackgroundAdvanced machine learning models have received wide attention in assisting medical decision making due to the greater accuracy they can achieve. However, their limited interpretability imposes barriers for practitioners to adopt them. Recent advancements in interpretable machine learning tools allow us to look inside the black box of advanced prediction methods to extract interpretable models while maintaining similar prediction accuracy, but few studies have investigated the specific hospital readmission prediction problem with this spirit.MethodsOur goal is to develop a machine-learning (ML) algorithm that can predict 30- and 90- day hospital readmissions as accurately as black box algorithms while providing medically interpretable insights into readmission risk factors. Leveraging a state-of-art interpretable ML model, we use a two-step Extracted Regression Tree approach to achieve this goal. In the first step, we train a black box prediction algorithm. In the second step, we extract a regression tree from the output of the black box algorithm that allows direct interpretation of medically relevant risk factors. We use data from a large teaching hospital in Asia to learn the ML model and verify our two-step approach.ResultsThe two-step method can obtain similar prediction performance as the best black box model, such as Neural Networks, measured by three metrics: accuracy, the Area Under the Curve (AUC) and the Area Under the Precision-Recall Curve (AUPRC), while maintaining interpretability. Further, to examine whether the prediction results match the known medical insights (i.e., the model is truly interpretable and produces reasonable results), we show that key readmission risk factors extracted by the two-step approach are consistent with those found in the medical literature.ConclusionsThe proposed two-step approach yields meaningful prediction results that are both accurate and interpretable. This study suggests a viable means to improve the trust of machine learning based models in clinical practice for predicting readmissions through the two-step approach.

Diabetic retinopathy lesion segmentation using deep multi-scale framework

The segmentation of diabetic retinopathy (DR) lesions is important for large-scale screening using color fundus photography (CFP) images. The difficulty of this task is that the DR lesions have various sizes, shapes, and intensities. Traditional handcrafted feature-based approaches are unsatisfactory, and recent deep-learning-based approaches ignore the features of DR lesions. In this paper, we propose a segmentation approach that segments four typical DR lesions simultaneously based on convolutional neural networks (CNN). The raw CFP image is first pre-processed and resized to different sizes. A set of fully convolutional neural networks (FCN) with various input sizes are then trained to extract the lesions from different scales. An auxiliary CNN is then introduced to fuse the output from these FCN and refine the segmentation result. We conducted our experiments on one local dataset and four public datasets including IDRiD, DDR, E-ophtha, and DIARETDB1. The results of the area under the precision–recall-curve (AUPR) and the dice similarity coefficient (DSC) show that our approach achieves competitive performance. The improvement in performance indicates that this approach is beneficial to DR lesion segmentation and has potential in other segmentation tasks.

Predictive value of clinician impression for readmission and postdischarge mortality among neonates and young children in Dar es Salaam, Tanzania and Monrovia, Liberia

BackgroundThere are no validated clinical decision aids to identify neonates and young children at risk of hospital readmission or postdischarge mortality in sub-Saharan Africa, leaving the decision to discharge a...

One way to provide effective healthcare services for patients with advanced cancer to improve their quality of life and survival: Prediction modeling using machine learning techniques and causal relationship between self-management strategy, health-related quality of life, and survival.

e24005 Background: In healthcare services, self-management strategies can play a crucial role in enhancing the health-related quality of life (HRQoL) or survival of cancer patients, regardless of their stage or treatment plan. However, there needs to be more research addressing the relationship between the strategies, HRQoL, and survival. To address this gap, we investigated the relationship between the strategies, HRQoL, and survival using machine learning techniques (MLT) and path analysis with structural equation modeling (SEM). Methods: This study was a secondary analysis of a randomized controlled trial (RCT) to provide early palliative care to individuals with advanced cancer (ClinicalTrials.gov, ID NCT03181854). We employed dimensional multiple Cox proportional hazard regression analyses to identify the crucial predictors of 1-year survival. By integrating the critical predictors extracted from the Cox regression models, we established a final survival prediction model using the eXtreme Gradient Boost (XGBoost) and the BorutaSHAP in MLT. Furthermore, we used path analysis to investigate the causal relationship between the strategies, HRQoL, and survival. Results: The univariate dimensional Cox models indicated that factors such as worse ECOG performance status (hazard ratio (HR) = 2.18), living alone (HR = 1.74), male (HR = 1.65), low global QoL (HR = 1.81), high dyspnea (HR = 2.30), high pain (HR = 1.75), high appetite loss (HR = 1.94), high constipation (HR = 2.11), high depression (HR = 2.79) at baseline, and no significant positive changes in emotional functioning (HR = 1.88) negatively impacted survival. The final XGBoost model for survival prediction using repeated K-fold methods (K = 5, repeats = 20, area under the receiver operating characteristic curve (AUROC) = 0.76, area under the precision-recall curve (AUPRC) = 0.87) and bootstrapping (n = 300, AUROC = 0.81, AUPRC = 0.91) showed high survival prediction performance. Additionally, the path analysis revealed a causal relationship between self-management strategies, HRQoL, and survival in patients with advanced cancer ( p-value = 0.005). HRQoL was found to have a complete mediating relationship between the strategies and survival. Self-management strategies did not directly correlate with survival but showed an impact on survival through HRQoL. Conclusions: To the best of our knowledge, this is the first study to uncover the connection between self-management strategies, HRQoL, and survival in patients with advanced cancer using MLT and path analysis. These findings have the potential to deepen our understanding of self-management strategies and aid healthcare providers in providing improved cancer care for patients with advanced cancer. Clinical trial information: NCT03181854 .

Radiofrequency ablation of lung metastases of colorectal cancer: could early radiomics analysis of the ablation zone help detect local tumor progression?

To determine whether radiomics data can predict local tumor progression (LTP) following radiofrequency ablation (RFA) of colorectal cancer (CRC) lung metastases on the first revaluation chest CT. This case-control single-center retrospective study included 95 distinct lung metastases treated by RFA (in 39 patients, median age: 63.1 years) with a contrast-enhanced CT-scan performed 3 months after RFA. Forty-eight radiomics features (RFs) were extracted from the 3D-segmentation of the ablation zone. Several supervised machine-learning algorithms were trained in 10-fold cross-validation on reproducible RFs to predict LTP, with/without denoising CT-scans. An unsupervised classification based on reproducible RFs was built with k-means algorithm. There were 20/95 (26.7%) relapses within a median delay of 10 months. The best model was a stepwise logistic regression on raw CT-scans. Its cross-validated performances were: AUROC = 0.72 (0.58-0.86), area under the Precision-Recall curve (AUPRC) = 0.44. Cross-validated balanced-accuracy, sensitivity and specificity were 0.59, 0.25 and 0.93, respectively, using p = 0.5 to dichotomize the model predicted probabilities (vs 0.71, 0.70 and 0.72, respectively using p = 0.188 according to Youden index). The unsupervised approach identified two clusters, which were not associated with LTP (p = 0.8211) but with the occurrence of per-RFA intra-alveolar hemorrhage, post-RFA cavitations and fistulizations (p = 0.0150). Predictive models using RFs from the post-RFA ablation zone on the first revaluation CT-scan of CRC lung metastases seemed moderately informative regarding the occurrence of LTP. Radiomics approach on interventional radiology data is feasible. However, patterns of heterogeneity detected with RFs on early re-evaluation CT-scans seem biased by different healing processes following benign RFA complications.

MultiDS-MDA: Integrating multiple data sources into heterogeneous network for predicting novel metabolite-drug associations

Metabolic processes in the human body play an important role in maintaining normal life activities, and the abnormal concentration of metabolites is closely related to the occurrence and development of diseases. The use of drugs is considered to have a major impact on metabolism, and drug metabolites can contribute to efficacy, drug toxicity and drug-drug interaction. However, our understanding of metabolite-drug associations is far from complete, and individual data source tends to be incomplete and noisy. Therefore, the integration of various types of data sources for inferring reliable metabolite-drug associations is urgently needed. In this study, we proposed a computational framework, MultiDS-MDA, for identifying metabolite-drug associations by integrating multiple data sources, including chemical structure information of metabolites and drugs, the relationships of metabolite-gene, metabolite-disease, drug-gene and drug-disease, the data of gene ontology (GO) and disease ontology (DO) and known metabolite-drug connections. The performance of MultiDS-MDA was evaluated by 5-fold cross-validation, which achieved an area under the ROC curve (AUROC) of 0.911 and an area under the precision-recall curve (AUPRC) of 0.907. Additionally, MultiDS-MDA showed outstanding performance compared with similar approaches. Case studies for three metabolites (cholesterol, thromboxane B2 and coenzyme Q10) and three drugs (simvastatin, pravastatin and morphine) also demonstrated the reliability and efficiency of MultiDS-MDA, and it is anticipated that MultiDS-MDA will serve as a powerful tool for future exploration of metabolite-drug interactions and contribute to drug development and drug combination.

Applying machine learning techniques to predict the risk of lung metastases from rectal cancer: a real-world retrospective study.

Metastasis in the lungs is common in patients with rectal cancer, and it can have severe consequences on their survival and quality of life. Therefore, it is essential to identify patients who may be at risk of developing lung metastasis from rectal cancer. In this study, we utilized eight machine-learning methods to create a model for predicting the risk of lung metastasis in patients with rectal cancer. Our cohort consisted of 27,180 rectal cancer patients selected from the Surveillance, Epidemiology and End Results (SEER) database between 2010 and 2017 for model development. Additionally, we validated our models using 1118 rectal cancer patients from a Chinese hospital to evaluate model performance and generalizability. We assessed our models' performance using various metrics, including the area under the curve (AUC), the area under the precision-recall curve (AUPR), the Matthews Correlation Coefficient (MCC), decision curve analysis (DCA), and calibration curves. Finally, we applied the best model to develop a web-based calculator for predicting the risk of lung metastasis in patients with rectal cancer. Our study employed tenfold cross-validation to assess the performance of eight machine-learning models for predicting the risk of lung metastasis in patients with rectal cancer. The AUC values ranged from 0.73 to 0.96 in the training set, with the extreme gradient boosting (XGB) model achieving the highest AUC value of 0.96. Moreover, the XGB model obtained the best AUPR and MCC in the training set, reaching 0.98 and 0.88, respectively. We found that the XGB model demonstrated the best predictive power, achieving an AUC of 0.87, an AUPR of 0.60, an accuracy of 0.92, and a sensitivity of 0.93 in the internal test set. Furthermore, the XGB model was evaluated in the external test set and achieved an AUC of 0.91, an AUPR of 0.63, an accuracy of 0.93, a sensitivity of 0.92, and a specificity of 0.93. The XGB model obtained the highest MCC in the internal test set and external validation set, with 0.61 and 0.68, respectively. Based on the DCA and calibration curve analysis, the XGB model had better clinical decision-making ability and predictive power than the other seven models. Lastly, we developed an online web calculator using the XGB model to assist doctors in making informed decisions and to facilitate the model's wider adoption (https://share.streamlit.io/woshiwz/rectal_cancer/main/lung.py). In this study, we developed an XGB model based on clinicopathological information to predict the risk of lung metastasis in patients with rectal cancer, which may help physicians make clinical decisions.

Comparative Study of C-TIRADS, ACR-TIRADS, and EU-TIRADS for Diagnosis and Management of Thyroid Nodules

Chinese Thyroid Imaging Reporting and Data Systems (C-TIRADS) was developed to provide a more simplified tool for stratifying thyroid nodules. Here we aimed to validate the efficacy of C-TIRADS in distinguishing benign from malignant and in guiding fine-needle aspiration biopsies in comparison with the American College of Radiology TIRADS (ACR-TIRADS) and European TIRADS (EU-TIRADS). This study retrospectively included 3438 thyroid nodules (≥10mm) in 3013 patients (mean age, 47.1 years±12.9) diagnosed between January 2013 and November 2019. Ultrasound features of the nodules were evaluated and categorized according to the lexicons of the three TIRADS. We compared these TIRADS by using the area under the receiver operating characteristic curve (AUROC), the area under the precision-recall curve (AUPRC), sensitivity, specificity, net reclassification improvement (NRI), and unnecessary fine-needle aspiration biopsy (FNAB) rate. Of the 3438 thyroid nodules, 707 (20.6%) were malignant. C-TIRADS showed higher discrimination performance (AUROC, 0.857; AUPRC, 0.605) than ACR-TIRADS (AUROC, 0.844; AUPRC, 0.567) and EU-TIRADS (AUROC, 0.802; AUPRC, 0.455). The sensitivity of C-TIRADS (85.3%) was lower than that of ACR-TIRADS (89.1%) but higher than that of EU-TIRADS (78.4%). The specificity of C-TIRADS (76.9%) was similar to that of EU-TIRADS (78.9%) and higher than that of ACR-TIRADS (69.5%). The unnecessary FNAB rate was lowest with C-TIRADS (21.2%), followed by ACR-TIRADS (41.7%) and EU-TIRADS (58.3%). C-TIRADS obtained significant NRI for recommending FNAB over ACR-TIRADS (19.0%, P<0.001) and EU-TIRADS (25.5%, P<0.001). C-TIRADS may be a clinically applicable tool to manage thyroid nodules, which warrants thorough tests in other geographic settings.

Development, multi-institutional external validation, and algorithmic audit of an artificial intelligence-based Side-specific Extra-Prostatic Extension Risk Assessment tool (SEPERA) for patients undergoing radical prostatectomy: a retrospective cohort study

Accurate prediction of side-specific extraprostatic extension (ssEPE) is essential for performing nerve-sparing surgery to mitigate treatment-related side-effects such as impotence and incontinence in patients with localised prostate cancer. Artificial intelligence (AI) might provide robust and personalised ssEPE predictions to better inform nerve-sparing strategy during radical prostatectomy. We aimed to develop, externally validate, and perform an algorithmic audit of an AI-based Side-specific Extra-Prostatic Extension Risk Assessment tool (SEPERA). Each prostatic lobe was treated as an individual case such that each patient contributed two cases to the overall cohort. SEPERA was trained on 1022 cases from a community hospital network (Trillium Health Partners; Mississauga, ON, Canada) between 2010 and 2020. Subsequently, SEPERA was externally validated on 3914 cases across three academic centres: Princess Margaret Cancer Centre (Toronto, ON, Canada) from 2008 to 2020; L'Institut Mutualiste Montsouris (Paris, France) from 2010 to 2020; and Jules Bordet Institute (Brussels, Belgium) from 2015 to 2020. Model performance was characterised by area under the receiver operating characteristic curve (AUROC), area under the precision recall curve (AUPRC), calibration, and net benefit. SEPERA was compared against contemporary nomograms (ie, Sayyid nomogram, Soeterik nomogram [non-MRI and MRI]), as well as a separate logistic regression model using the same variables included in SEPERA. An algorithmic audit was performed to assess model bias and identify common patient characteristics among predictive errors. Overall, 2468 patients comprising 4936 cases (ie, prostatic lobes) were included in this study. SEPERA was well calibrated and had the best performance across all validation cohorts (pooled AUROC of 0·77 [95% CI 0·75-0·78] and pooled AUPRC of 0·61 [0·58-0·63]). In patients with pathological ssEPE despite benign ipsilateral biopsies, SEPERA correctly predicted ssEPE in 72 (68%) of 106 cases compared with the other models (47 [44%] in the logistic regression model, none in the Sayyid model, 13 [12%] in the Soeterik non-MRI model, and five [5%] in the Soeterik MRI model). SEPERA had higher net benefit than the other models to predict ssEPE, enabling more patients to safely undergo nerve-sparing. In the algorithmic audit, no evidence of model bias was observed, with no significant difference in AUROC when stratified by race, biopsy year, age, biopsy type (systematic only vs systematic and MRI-targeted biopsy), biopsy location (academic vs community), and D'Amico risk group. According to the audit, the most common errors were false positives, particularly for older patients with high-risk disease. No aggressive tumours (ie, grade >2 or high-risk disease) were found among false negatives. We demonstrated the accuracy, safety, and generalisability of using SEPERA to personalise nerve-sparing approaches during radical prostatectomy. None.

Threshold optimization and random undersampling for imbalanced credit card data

Output thresholding is well-suited for addressing class imbalance, since the technique does not increase dataset size, run the risk of discarding important instances, or modify an existing learner. Through the use of the Credit Card Fraud Detection Dataset, this study proposes a threshold optimization approach that factors in the constraint True Positive Rate (TPR) ≥ True Negative Rate (TNR). Our findings indicate that an increase of the Area Under the Precision–Recall Curve (AUPRC) score is associated with an improvement in threshold-based classification scores, while an increase of positive class prior probability causes optimal thresholds to increase. In addition, we discovered that best overall results for the selection of an optimal threshold are obtained without the use of Random Undersampling (RUS). Furthermore, with the exception of AUPRC, we established that the default threshold yields good performance scores at a balanced class ratio. Our evaluation of four threshold optimization techniques, eight threshold-dependent metrics, and two threshold-agnostic metrics defines the uniqueness of this research.

Dynamic prediction of hypoxemia risk at different time points based on preoperative and intraoperative features: machine learning applications in outpatients undergoing esophagogastroduodenoscopy

Background Hypoxemia often occurs in outpatients undergoing anesthesia-assisted esophagogastroduodenoscopy (EGD). However, there is a scarcity in tools to predict the hypoxemia risk. We aimed to solve this problem by developing and validating machine learning (ML) models based on preoperative and intraoperative features. Methods All data were retrospectively collected from June 2021 to February 2022. The most appropriate predictive features were selected by the least absolute shrinkage and selection operator, which were incorporated and modelled by 4 ML algorithms. The area under the precision-recall curve (AUPRC) was used as the main evaluation metric to select the best models, and the selected models were compared with the STOP-BANG score. Their predictive performance was visually interpreted by SHapley Additive exPlanations. The primary endpoint of this study was hypoxemia during the procedure, defined as at least one reading of pulse oximetry < 90% without probes misplacement from the anesthesia induction beginning to the end of EGD, while the secondary endpoint was hypoxemia during induction, from the induction beginning to the start of endoscopic intubation. Results Of 1160 patients in the derivation cohort, 112 patients (9.6%) developed intraoperative hypoxemia, of which 102 (8.8%) occurred during the induction period. In temporal and external validation, no matter whether based on preoperative variables or still based on preoperative plus intraoperative variables, our models showed excellent predictive performance for the two endpoints, significantly better than STOP-BANG score. In the model interpretation section, preoperative variables (airway assessment indicators, pulse oximeter oxygen saturation and BMI) and intraoperative variables (the induced propofol dose) made the highest contribution to the predictions. Conclusions To our knowledge, our ML models were the first to predict hypoxemia risk, which achieved excellent overall predictive ability integrating various clinical indicators. These models have the potential to become an effective tool for adjusting sedation strategies flexibly and reducing the workload of anesthesiologists. KEY MESSAGES This study is the first model employing ML methods based on preoperative and preoperative plus intraoperative variables for predicting the risk of hypoxemia during induction and the whole EGD procedure respectively. Our four models achieved satisfactory predictive performance and outperformed STOP-BANG score in terms of AUPRC in the temporal and external validation cohorts respectively. We found that the relevant variables of airway assessment should be fully taken into account when analyzing the risk factor of hypoxemia, and the effect of patients’ age on their hypoxemia risk should be considered in conjunction with the propofol dose.

A natural language processing approach to categorise contributing factors from patient safety event reports

ObjectivesThe objective of this study was to explore the use of natural language processing (NLP) algorithm to categorise contributing factors from patient safety event (PSE). Contributing factors are elements in...

Machine Learning Prediction of Seizures after Ischemic Strokes. (S30.007)

<h3>Objective:</h3> To develop an artificial intelligence, machine learning prediction model for estimating the risk of seizures at 1 year and 5 years after ischemic stroke (IS). <h3>Background:</h3> Seizures are frequent complications of stroke and are associated with increased mortality, poor functional outcome, and lower quality of life. There are however no established clinical approaches to identify stroke survivors who are at high risk for seizures. <h3>Design/Methods:</h3> We identified patients with IS without a prior diagnosis of seizures from 2015 until inception (08/09/22) in the TriNetX Diamond Network, using the International Classification of Diseases, Tenth Revision (ICD-10) I63, excluding I63.6. The outcome of interest was any ICD-10 diagnosis of seizures (G40/G41) at 1 year and 5 years following the index IS. We applied a conventional logistic regression and a Light Gradient Boosted Machine (LGBM) algorithm to predict the risk of seizures at 1 year and 5 years. The performance of the model was assessed using the area under the receiver operating characteristics (AUROC), the area under the precision-recall curve (AUPRC), the F1 statistic, model accuracy, balanced-accuracy, precision, and recall, with and without seizure medication use in the models. <h3>Results:</h3> We included 444,130 IS patients without a prior diagnosis of seizures. Seizures were present in 20,085 (4.2%) and 24,479 (5.5%) patients within 1 and 5 years after IS, respectively. At 1-year, the AUROC, AUPRC, F1 statistic, accuracy, balanced-accuracy, precision, and recall were respectively 0.7917 (standard error: 0.0029), 0.2604 (0.003), 0.238 (0.0016), 0.819 (0.0024), 0.7267 (0.0008), 0.147 (0.0014), and 0.6252 (0.0036). Corresponding metrics at 5 years were 0.767 (0.0042), 0.2633 (0.0087), 0.2535 (0.0057), 0.8087 (0.0038), 0.7053 (0.0052), 0.1615 (0.0042) and 0.5892 (0.0074). <h3>Conclusions:</h3> Our findings suggest that ML models show good model performance for predicting seizures after IS. Incorporating imaging variables, including markers of brain health could improve these models. <b>Disclosure:</b> Dr. Lekoubou Looti has nothing to disclose. Dr. Petucci has nothing to disclose. Mr. Katoch has nothing to disclose. The institution of Dr. Honavar has received research support from National Science Foundation. The institution of Dr. Honavar has received research support from National Institutes of Health. The institution of Dr. Honavar has received research support from Volvo. The institution of Dr. Honavar has received research support from Department of Homeland Security. The institution of Dr. Honavar has received research support from Samsung . Dr. Honavar has received personal compensation in the range of $0-$499 for serving as a Review Panelist with National Science Foundation. Dr. Honavar has received personal compensation in the range of $0-$499 for serving as a Study Section Member with National Institutes of Health.

Development and assessment of novel machine learning models to predict the probability of postoperative nausea and vomiting for patient-controlled analgesia

Postoperative nausea and vomiting (PONV) can lead to various postoperative complications. The risk assessment model of PONV is helpful in guiding treatment and reducing the incidence of PONV, whereas the published models of PONV do not have a high accuracy rate. This study aimed to collect data from patients in Sichuan Provincial People’s Hospital to develop models for predicting PONV based on machine learning algorithms, and to evaluate the predictive performance of the models using the area under the receiver characteristic curve (AUC), accuracy, precision, recall rate, F1 value and area under the precision-recall curve (AUPRC). The AUC (0.947) of our best machine learning model was significantly higher than that of the past models. The best of these models was used for external validation on patients from Chengdu First People’s Hospital, and the AUC was 0.821. The contributions of variables were also interpreted using SHapley Additive ExPlanation (SHAP). A history of motion sickness and/or PONV, sex, weight, history of surgery, infusion volume, intraoperative urine volume, age, BMI, height, and PCA_3.0 were the top ten most important variables for the model. The machine learning models of PONV provided a good preoperative prediction of PONV for intravenous patient-controlled analgesia.

An interpretable diagnostic approach for lung cancer: Combining maximal clique and improved BERT

AbstractThe lung cancer incidence and mortality in China have always been high. Moreover, due to the limited level of professional technology, misdiagnosis and missed diagnosis of lung cancer often occur. To improve the accuracy of diagnosis, this paper proposes an interpretable diagnostic method for lung cancer based on Chinese electronic medical records (EMRs). First, to overcome the difficulty in word segmentation of clinical texts in Chinese EMRs, a dictionary construction method is proposed based on the idea of maximal clique, and 730 medical professional terms related to lung diseases are identified. Then, the ProbSparse self‐attention mechanism and self‐attention distilling operation in Informer are used to improve the Bidirectional Encoder Representations from Transformer (BERT) to realize the representation of long clinical texts with lower time complexity and memory consumption. Finally, the convolutional neural network with an attention mechanism is employed to process the representation results to realize the interpretable prediction of lung cancer. This method is applied to the lung cancer diagnosis of inpatients in a tertiary hospital in Hunan Province, obtaining excellent results of about 0.9 for area under the receiver operating characteristic curve (AUROC) and area under the precision‐recall curve (AUPRC). In addition, the results of the comparative analysis with existing dictionaries, word embedding methods and diagnostic methods further confirm the superiority of the proposed method. Specifically, the proposed method improves the precision by at least 6%, the recall by at least 2.6%, the F1 score by at least 5.2%, AUROC by at least 7.3% and AUPRC by at least 7.7% compared with all these state‐of‐the‐art methods.

Early prediction of delirium upon intensive care unit admission: Model development, validation, and deployment

Study objectiveTo develop, validate, and deploy models for predicting delirium in critically ill adult patients as early as upon intensive care unit (ICU) admission. DesignRetrospective cohort study. SettingSingle university teaching hospital in Taipei, Taiwan. Patients6238 critically ill patients from August 2020 to August 2021. MeasurementsData were extracted, pre-processed, and split into training and testing datasets based on the time period. Eligible variables included demographic characteristics, Glasgow Coma Scale, vital signs parameters, treatments, and laboratory data. The predicted outcome was delirium, defined as any positive result (a score ≥ 4) of the Intensive Care Delirium Screening Checklist that was assessed by primary care nurses in each 8-h shift within 48 h after ICU admission. We trained models to predict delirium upon ICU admission (ADM) and at 24 h (24H) after ICU admission by using logistic regression (LR), gradient boosted trees (GBT), and deep learning (DL) algorithms and compared the models' performance. Main resultsEight features were extracted from the eligible features to train the ADM models, including age, body mass index, medical history of dementia, postoperative intensive monitoring, elective surgery, pre-ICU hospital stays, and GCS score and initial respiratory rate upon ICU admission. In the ADM testing dataset, the incidence of ICU delirium occurred within 24 h and 48 h was 32.9% and 36.2%, respectively. The area under the receiver operating characteristic curve (AUROC) (0.858, 95% CI 0.835–0.879) and area under the precision-recall curve (AUPRC) (0.814, 95% CI 0.780–0.844) for the ADM GBT model were the highest. The Brier scores of the ADM LR, GBT, and DL models were 0.149, 0.140, and 0.145, respectively. The AUROC (0.931, 95% CI 0.911–0.949) was the highest for the 24H DL model and the AUPRC (0.842, 95% CI 0.792–0.886) was the highest for the 24H LR model. ConclusionOur early prediction models based on data obtained upon ICU admission could achieve good performance in predicting delirium occurred within 48 h after ICU admission. Our 24-h models can improve delirium prediction for patients discharged >1 day after ICU admission.

Evaluating classifier performance with highly imbalanced Big Data

Using the wrong metrics to gauge classification of highly imbalanced Big Data may hide important information in experimental results. However, we find that analysis of metrics for performance evaluation and what they can hide or reveal is rarely covered in related works. Therefore, we address that gap by analyzing multiple popular performance metrics on three Big Data classification tasks. To the best of our knowledge, we are the first to utilize three new Medicare insurance claims datasets which became publicly available in 2021. These datasets are all highly imbalanced. Furthermore, the datasets are comprised of completely different data. We evaluate the performance of five ensemble learners in the Machine Learning task of Medicare fraud detection. Random Undersampling (RUS) is applied to induce five class ratios. The classifiers are evaluated with both the Area Under the Receiver Operating Characteristic Curve (AUC), and Area Under the Precision Recall Curve (AUPRC) metrics. We show that AUPRC provides a better insight into classification performance. Our findings reveal that the AUC metric hides the performance impact of RUS. However, classification results in terms of AUPRC show RUS has a detrimental effect. We show that, for highly imbalanced Big Data, the AUC metric fails to capture information about precision scores and false positive counts that the AUPRC metric reveals. Our contribution is to show AUPRC is a more effective metric for evaluating the performance of classifiers when working with highly imbalanced Big Data.

A deep neural network framework to derive interpretable decision rules for accurate traumatic brain injury identification of infants

ObjectiveWe aimed to develop a robust framework to model the complex association between clinical features and traumatic brain injury (TBI) risk in children under age two, and identify significant features to derive clinical decision rules for triage decisions.MethodsIn this retrospective study, four frequently used machine learning models, i.e., support vector machine (SVM), random forest (RF), deep neural network (DNN), and XGBoost (XGB), were compared to identify significant clinical features from 24 input features associated with the TBI risk in children under age two under the permutation feature importance test (PermFIT) framework by using the publicly available data set from the Pediatric Emergency Care Applied Research Network (PECARN) study. The prediction accuracy was determined by comparing the predicted TBI status with the computed tomography (CT) scan results since CT scan is the gold standard for diagnosing TBI.ResultsAt a significance level of p = 0.05, DNN, RF, XGB, and SVM identified 9, 1, 2, and 4 significant features, respectively. In a comparison of accuracy (Accuracy), the area under the curve (AUC), and the precision-recall area under the curve (PR-AUC), the permutation feature importance test for DNN model was the most powerful framework for identifying significant features and outperformed other methods, i.e., RF, XGB, and SVM, with Accuracy, AUC, and PR-AUC as 0.915, 0.794, and 0.974, respectively.ConclusionThese results indicate that the PermFIT-DNN framework robustly identifies significant clinical features associated with TBI status and improves prediction performance. The findings could be used to inform the development of clinical decision tools designed to inform triage decisions.

Development and validation of DNA methylation scores in two European cohorts augment 10-year risk prediction of type 2 diabetes.

Type 2 diabetes mellitus (T2D) presents a major health and economic burden that could be alleviated with improved early prediction and intervention. While standard risk factors have shown good predictive performance, we show that the use of blood-based DNA methylation information leads to a significant improvement in the prediction of 10-year T2D incidence risk. Previous studies have been largely constrained by linear assumptions, the use of cytosine-guanine pairs one-at-a-time and binary outcomes. We present a flexible approach (via an R package, MethylPipeR) based on a range of linear and tree-ensemble models that incorporate time-to-event data for prediction. Using the Generation Scotland cohort (training set ncases = 374, ncontrols = 9,461; test set ncases = 252, ncontrols = 4,526) our best-performing model (area under the receiver operating characteristic curve (AUC) = 0.872, area under the precision-recall curve (PRAUC) = 0.302) showed notable improvement in 10-year onset prediction beyond standard risk factors (AUC = 0.839, precision-recall AUC = 0.227). Replication was observed in the German-based KORA study (n = 1,451, ncases = 142, P = 1.6 × 10-5).

An Interpretable Machine Learning Model for Predicting 10-Year Total Hip Arthroplasty Risk

BackgroundAs the demand for total hip arthroplasty (THA) rises, a predictive model for THA risk may aid patients and clinicians in augmenting shared decision-making. We aimed to develop and validate a model predicting THA within 10 years in patients using demographic, clinical, and deep learning (DL)-automated radiographic measurements. MethodsPatients enrolled in the osteoarthritis initiative were included. DL algorithms measuring osteoarthritis- and dysplasia-relevant parameters on baseline pelvis radiographs were developed. Demographic, clinical, and radiographic measurement variables were then used to train generalized additive models to predict THA within 10 years from baseline. A total of 4,796 patients were included [9,592 hips; 58% female; 230 THAs (2.4%)]. Model performance using 1) baseline demographic and clinical variables 2) radiographic variables, and 3) all variables was compared. ResultsUsing 110 demographic and clinical variables, the model had a baseline area under the receiver operating curve (AUROC) of 0.68 and area under the precision recall curve (AUPRC) of 0.08. Using 26 DL-automated hip measurements, the AUROC was 0.77 and AUPRC was 0.22. Combining all variables, the model improved to an AUROC of 0.81 and AUPRC of 0.28. Three of the top five predictive features in the combined model were radiographic variables, including minimum joint space, along with hip pain and analgesic use. Partial dependency plots revealed predictive discontinuities for radiographic measurements consistent with literature thresholds of osteoarthritis progression and hip dysplasia. ConclusionA machine learning model predicting 10-year THA performed more accurately with DL radiographic measurements. The model weighted predictive variables in concordance with clinical THA pathology assessments.