Area Under The Curve Of Model Research Articles

Pediatric prognostic models predicting inhospital child mortality in resource-limited settings: An external validation study.

Prognostic models provide evidence-based predictions and estimates of future outcomes, facilitating decision-making, patient care, and research. A few of these models have been externally validated, leading to uncertain reliability and generalizability. This study aims to externally validate four models to assess their transferability and usefulness in clinical practice. The models include the respiratory index of severity in children(RISC)-Malawi model and three other models by Lowlavaar et al. The study used data from the Clinical Information Network (CIN) to validate the four models where the primary outcome was in-hospital mortality. 163,329 patients met eligibility criteria. Missing data were imputed, and the logistic function was used to compute predicted risk of in-hospital mortality. Models' discriminatory ability and calibration were determined using area under the curve(AUC), calibration slope, and intercept. The RISC-Malawi model had 50,669 pneumonia patients who met the eligibility criteria, of which the case-fatality ratio was 4406 (8.7%). Its AUCwas 0.77 (95% CI: 0.77-0.78), whereas the calibration slope was 1.04 (95% CI: 1.00 -1.06), and calibration intercept was 0.81 (95% CI: 0.77-0.84). Regarding the external validation of Lowlavaar et al. models, 10,782 eligible patients were included, with an in-hospital mortality rate of 5.3%. The primary model's AUC was 0.75 (95% CI: 0.72-0.77), the calibration slope was 0.78 (95% CI: 0.71-0.84), and the calibration intercept was 0.37 (95% CI: 0.28-0.46). All models markedly underestimated the risk of mortality. All externally validated models exhibited either underestimation or overestimation of the risk as judged from calibration statistics. Hence, applying these models with confidence in settings other than their original development context may not be advisable. Our findings strongly suggest the need for recalibrating these model to enhance their generalizability.

Machine Learning-Enabled 30-Day Readmission Model for Stroke Patients.

Background and Purpose: Hospital readmissions impose a substantial burden on the healthcare system. Reducing readmissions after stroke could lead to improved quality of care especially since stroke is associated with a high rate of readmission. The goal of this study is to enhance our understanding of the predictors of 30-day readmission after ischemic stroke and develop models to identify high-risk individuals for targeted interventions.Methods: We used patient-level data from electronic health records (EHR), five machine learning algorithms (random forest, gradient boosting machine, extreme gradient boosting–XGBoost, support vector machine, and logistic regression-LR), data-driven feature selection strategy, and adaptive sampling to develop 15 models of 30-day readmission after ischemic stroke. We further identified important clinical variables.Results: We included 3,184 patients with ischemic stroke (mean age: 71 ± 13.90 years, men: 51.06%). Among the 61 clinical variables included in the model, the National Institutes of Health Stroke Scale score above 24, insert indwelling urinary catheter, hypercoagulable state, and percutaneous gastrostomy had the highest importance score. The Model's AUC (area under the curve) for predicting 30-day readmission was 0.74 (95%CI: 0.64–0.78) with PPV of 0.43 when the XGBoost algorithm was used with ROSE-sampling. The balance between specificity and sensitivity improved through the sampling strategy. The best sensitivity was achieved with LR when optimized with feature selection and ROSE-sampling (AUC: 0.64, sensitivity: 0.53, specificity: 0.69).Conclusions: Machine learning-based models can be designed to predict 30-day readmission after stroke using structured data from EHR. Among the algorithms analyzed, XGBoost with ROSE-sampling had the best performance in terms of AUC while LR with ROSE-sampling and feature selection had the best sensitivity. Clinical variables highly associated with 30-day readmission could be targeted for personalized interventions. Depending on healthcare systems' resources and criteria, models with optimized performance metrics can be implemented to improve outcomes.

Predicting Response to Hypomethylating Agents in Patients with Myelodysplastic Syndromes (MDS) Using Artificial Intelligence (AI)

Introduction While the hypomethylating agents (HMAs) azacitidine (AZA) and decitabine (DAC) improve cytopenias and prolong survival in MDS patients (pts), response is not guaranteed. Timely identification of non-responders could prevent prolonged exposure to ineffective therapy, thereby reducing toxicities and costs. Currently no widely accepted clinical or genomic models exist to predict response or resistance to HMAs. We developed a clinical model to predict response or resistance to HMA after 90 days of initiating therapy based on changes in blood counts using time series analysis technology similar to the kind used in Apple's Siri or Google Assistant. In the setting of voice recognition, the sequence and context of words determines the meaning of a sentence; similarly, we hypothesized that the pattern of changes in MDS pts' blood counts would predict response or resistance early during treatment. Methods We screened a cohort of 107 pts with MDS (per 2016 WHO criteria) who received HMAs at our institution between February 2005 and July 2013 and had regular CBCs drawn during treatment. Mutations from a panel of 60 genes commonly mutated in myeloid malignancy were included. Responses were assessed after 6 months of therapy per International Working Group (IWG) 2006 criteria. Pts were divided randomly into training (80%) and validation (20%) cohorts. To address the potential for bias due to a small sample size, an oversampling algorithm was used to cluster similar pts based on their CBC data, Revised International Prognostic Scoring System (IPSS-R) score, and % bone marrow blasts at the time of diagnosis. CBC data from the first 90 days of treatment were fed into deep neural network (recurrent neural network) and decision tree algorithms, which were trained to predict whether pts would achieve a response (defined as complete remission (CR), partial remission (PR), or hematologic Improvement (HI)). Area under the curve (AUC) was used to assess model performance. Important features that impact the algorithm's predictions were extracted and plotted. Results 20747 unique data points were used, including CBC, clinical and genomic data. Among 107 pts, 61 (57.0%) received AZA only, 19 (17.8%) DAC only, 4 (3.7%) received both DAC and AZA, and 23 (21.5%) received HMA with an additional agent. Median age was 69 years (range: 37-100 years), and 27 (26.4%) were female. Forty pts (37.4%) were very low/low risk, 32 (29.9%) intermediate, 19 (17.8%) high, and 16 (14.9%) very high risk per IPSS-R. Responses included 23 (22.5%) CR, 2 (1.9%) marrow CR, 4 (3.9%) PR, and 20 (19.6%) HI. The most commonly mutated genes were ASXL1 (17.6%), TET2 (16.7%), SRSF2 (15.7%), SF3B1 (11.8%), RUNX1 (10.8%), STAG2(10.8%), and DNMT3A (10.8%). The median number of mutations per sample was 1 (range, 0-11), and 40 pts (39.2%) had > 3 mutations per sample. When trained using absolute values and changes in CBC values, the model's AUC was 0.95 in the training cohort and 0.83 in the validation cohort. When the cohort was oversampled to 1000 pts, the validation cohort AUC increased to 0.89. Feature extraction algorithms identified increases in MCV and RDW during weeks 2-8 of treatment, increased proportion of lymphocytes, decreased proportion of monocytes, and increased platelet counts during weeks 6-8 as factors favoring response to HMA. The model provides personalized, patient-specific predictions that correlate with blood counts (Figure 1). Conclusions We describe a machine learning model that monitors changes in blood counts during therapy with HMA to predict response or resistance to HMA in MDS pts. Such a model can be used to develop novel trial designs wherein pts predicted to not respond after 90 days of HMA treatment could be assigned to an investigational agent. Conversely, it would help inform the decision to continue HMA therapy in pts predicted to respond. Increasing sample size with oversampling dramatically increased model accuracy; a larger cohort of pts treated at different institutions is currently under development. Disclosures Sekeres: Millenium: Membership on an entity's Board of Directors or advisory committees; Syros: Membership on an entity's Board of Directors or advisory committees; Celgene: Membership on an entity's Board of Directors or advisory committees. Mukherjee:Partnership for Health Analytic Research, LLC (PHAR, LLC): Consultancy; Takeda: Membership on an entity's Board of Directors or advisory committees; Celgene Corporation: Consultancy, Membership on an entity's Board of Directors or advisory committees, Research Funding; Projects in Knowledge: Honoraria; Novartis: Consultancy, Membership on an entity's Board of Directors or advisory committees, Research Funding; Pfizer: Honoraria; McGraw Hill Hematology Oncology Board Review: Other: Editor; Bristol-Myers Squibb: Speakers Bureau. Advani:Glycomimetics: Consultancy, Research Funding; Kite Pharmaceuticals: Consultancy; Amgen: Research Funding; Pfizer: Honoraria, Research Funding; Macrogenics: Research Funding; Abbvie: Research Funding. Maciejewski:Alexion: Consultancy; Novartis: Consultancy. Nazha:Novartis: Speakers Bureau; Tolero, Karyopharma: Honoraria; Abbvie: Consultancy; Jazz Pharmacutical: Research Funding; Incyte: Speakers Bureau; Daiichi Sankyo: Consultancy; MEI: Other: Data monitoring Committee.

Classical sheep scrapie in Great Britain: spatial analysis and identification of environmental and farm-related risk factors

BackgroundPrevious studies suggest that the spatial distribution of classical sheep scrapie in Great Britain is uneven and that certain flock characteristics may be associated with occurrence of the disease. However, the existence of areas of high and low disease-risk may also result from differences in the spatial distribution of environmental characteristics. In this study we explored the spatial pattern of classical scrapie in Great Britain between 2002 and 2005 and investigated the association between disease occurrence and various environmental and farm-related risk factors.ResultsExploratory spatial analysis: South Wales was found to have a higher density of scrapie-positive farms than the rest of Great Britain. In addition, a small cluster of high-risk farms was identified in the center of this region in which clustering of scrapie-positive farms occurred up to a distance of approximately 40 km.Spatial modelling: A mixed-effects regression model identified flock-size and soil drainage to be significantly associated with the occurrence of scrapie in England and Wales (area under the curve (AUC) 0.71 ± 0.01, 95% CI 0.68 - 0.74). The predictive risk map based on the estimated association between these factors and disease occurrence showed most of Wales to be at risk of being confirmed positive for scrapie with areas of highest risk in central and south Wales. In England, areas with the highest risk occurred mainly in the north and the midlands.ConclusionThe observed distribution of scrapie in Great Britain exhibited a definite spatial pattern with south Wales identified as an area of high occurrence. In addition both flock (flock size) and environmental variables (soil drainage) were found to be significantly associated with the occurrence of the disease. However, the model's AUC indicated unexplained variation remaining in the model and the source of this variation may lie in farm-level characteristics rather than spatially-varying ones such as environmental factors.