Evaluated Model Discrimination Research Articles

Machine learning prediction for prognosis and long-term effectiveness for transcatheter ventricular septal defect closure: a 5-year single center experience

Abstract Background Transcatheter therapy, with minimal invasiveness and rapid recovery advantages, has emerged as the primary choice for treating perimembranous ventricular septal defect. However, patients receiving transcatheter occlusion may confront conduction abnormalities due to the occluder's proximity to the cardiac conduction bundle during implantation. Few have assessed the risks of postoperative conduction block (CB) via robust machine learning models. Purpose The purpose of this study was to evaluate our long-term follow up results of pmVSD occlusion and establish a prediction model to identify risk predictors of conduction abnormalities via robust machine learning methods. Methods Five methods including logistic regression with a stepwise selection of variables, lasso regularization; random forest; gradient descent boosting (GBM); and support vector machine, were used to train models for predicting risks of late-onset and persistent conduction block through 5 years of follow-up and were validated using 5-fold cross-validation. Receiver-operating characteristic curves and Brier scores were utilized to evaluate model discrimination and calibration, respectively. The top prediction variables were identified by using the best performing models, using the relative influence score of each variable in 5-fold cross-validation. Results The GBM model performed the best with an AUC of 0.82 (95% confidence interval [CI]: 0.75 to 0.89) for predicting late-onset CB (Brier score: 0.0204), and 0.70 (95% CI: 0.62 to 0.78) for persistent CB (Brier score: 0.0003) over entire study period. Conduction block on admission, advanced age and aortic regurgitation before discharge were strongly associated with late-onset CB, whereas decreased left ventricular ejection fraction, enlarged left ventricle end diastolic diameter at 1-month follow-up were the most significant predictors of persistent CB. Conclusions These models predict the risks of temporary and persistent postoperative conduction block in patients with pmVSD occlusion and shed light on preventing long-term complications.Study FlowchartCentral Illustration

Geographical validation of the Smart Triage Model by age group.

Infectious diseases in neonates account for half of the under-five mortality in low- and middle-income countries. Data-driven algorithms such as clinical prediction models can be used to efficiently detect critically ill children in order to optimize care and reduce mortality. Thus far, only a handful of prediction models have been externally validated and are limited to neonatal in-hospital mortality. The aim of this study is to externally validate a previously derived clinical prediction model (Smart Triage) using a combined prospective baseline cohort from Uganda and Kenya with a composite endpoint of hospital admission, mortality, and readmission. We evaluated model discrimination using area under the receiver-operator curve (AUROC) and visualized calibration plots with age subsets (< 30 days, ≤ 2 months, ≤ 6 months, and < 5 years). Due to reduced performance in neonates (< 1 month), we re-estimated the intercept and coefficients and selected new thresholds to maximize sensitivity and specificity. 11595 participants under the age of five (under-5) were included in the analysis. The proportion with an endpoint ranged from 8.9% in all children under-5 (including neonates) to 26% in the neonatal subset alone. The model achieved good discrimination for children under-5 with AUROC of 0.81 (95% CI: 0.79-0.82) but poor discrimination for neonates with AUROC of 0.62 (95% CI: 0.55-0.70). Sensitivity at the low-risk thresholds (CI) were 85% (83%-87%) and 68% (58%-76%) for children under-5 and neonates, respectively. After model revision for neonates, we achieved an AUROC of 0.83 (95% CI: 0.79-0.87) with 13% and 41% as the low- and high-risk thresholds, respectively. The updated Smart Triage performs well in its predictive ability across different age groups and can be incorporated into current triage guidelines at local healthcare facilities. Additional validation of the model is indicated, especially for the neonatal model.

Estimating risk for pancreatic cancer among 9.4 million veterans in care.

10544 Background: Screening for pancreatic ductal adenocarcinoma (PDAC) has focused on individuals with a hereditary component, accounting for only 10% of PDAC events. Using the electronic health record, we developed a PDAC risk prediction model in a general population in the Veterans Health Administration (VA), the largest integrated health system in the US. Methods: We included all individuals from 2002- 2021 with ≥1 inpatient or outpatient VA encounter and recorded race, starting follow-up 1.5 years after first VA encounter. We excluded those who developed PDAC before start of follow-up. We used the multivariable Cox proportional hazards model in the derivation dataset (random 70% of cohort) to estimate the coefficients associated with each predictor for our outcome, time to PDAC (3 and 10 years), censoring at loss to follow-up or 12/31/2021. Candidate predictors included age, gender, race/ethnicity, sex, BMI, smoking status, alcohol use (alcohol use/related disorders and AUDIT-C), diabetes, pancreatitis, Hepatitis C, HIV, liver disease, cholecystectomy, pancreatic cysts, Charlson comorbidity index and common laboratory values. Selection of predictors was based on clinical reasoning, extent of missing values and model fit (Akaike Information Criterion and Harrell’s c–statistic). We evaluated model discrimination using c – statistic and model calibration comparing observed Kaplan – Meier estimated incidence vs. predicted risk score. Results: Among 9.4 million individuals, we observed 19,839 cases of PDAC within 10 years (6,022 cases within 3 years). Median baseline age was 58.5 years, 75.0% were White, and 16.3% were Black. Final predictors included age, gender, smoking, alcohol use (alcohol use/related disorders and AUDIT-C), BMI, diabetes, pancreatitis, pancreatic cyst, and history of cancer (Table, c-statistic in the derivation cohort 3 years: 0.79; 10 years: 0.77). Ten-year risk ranged maximally from 0-5.5%. Preliminary studies suggest labs do not clinically improve model performance. Conclusions: We developed a novel prediction model using available clinical data in a general population that identifies PDAC with high accuracy. Future work will try to identify additional predictors to improve the range of risk predicted. We will validate our findings in the testing set and in external cohorts. [Table: see text]

A Clinical Prediction Model to Assess Risk for Pancreatic Cancer Among Patients With Acute Pancreatitis.

We aimed to develop and validate a prediction model as the first step in a sequential screening strategy to identify acute pancreatitis (AP) individuals at risk for pancreatic cancer (PC). We performed a population-based retrospective cohort study among individuals 40 years or older with a hospitalization for AP in the US Veterans Health Administration. For variable selection, we used least absolute shrinkage and selection operator regression with 10-fold cross-validation to identify a parsimonious logistic regression model for predicting the outcome, PC diagnosed within 2 years after AP. We evaluated model discrimination and calibration. Among 51,613 eligible study patients with AP, 801 individuals were diagnosed with PC within 2 years. The final model (area under the receiver operating curve, 0.70; 95% confidence interval, 0.67-0.73) included histories of gallstones, pancreatic cyst, alcohol use, smoking, and levels of bilirubin, triglycerides, alkaline phosphatase, aspartate aminotransferase, alanine aminotransferase, and albumin. If the predicted risk threshold was set at 2% over 2 years, 20.3% of the AP population would undergo definitive screening, identifying nearly 50% of PC associated with AP. We developed a prediction model using widely available clinical factors to identify high-risk patients with PC-associated AP, the first step in a sequential screening strategy.

Performance of risk prediction models for post-liver transplant patient and graft survival over time.

Given liver transplantation organ scarcity, selection of recipients and donors to maximize post-transplant benefit is paramount. Several scores predict post-transplant outcomes by isolating elements of donor and recipient risk, including the donor risk index, Balance of Risk, pre-allocation score to predict survival outcomes following liver transplantation/survival outcomes following liver transplantation (SOFT), improved donor-to-recipient allocation score for deceased donors only/improved donor-to-recipient allocation score for both deceased and living donors (ID2EAL-D/-DR), and survival benefit (SB) models. No studies have examined the performance of these models over time, which is critical in an ever-evolving transplant landscape. This was a retrospective cohort study of liver transplantation events in the UNOS database from 2002 to 2021. We used Cox regression to evaluate model discrimination (Harrell's C) and calibration (testing of calibration curves) for post-transplant patient and graft survival at specified post-transplant timepoints. Sub-analyses were performed in the modern transplant era (post-2014) and for key donor-recipient characteristics. A total of 112,357 transplants were included. The SB and SOFT scores had the highest discrimination for short-term patient and graft survival, including in the modern transplant era, where only the SB model had good discrimination (C ≥ 0.60) for all patient and graft outcome timepoints. However, these models had evidence of poor calibration at 3- and 5-year patient survival timepoints. The ID2EAL-DR score had lower discrimination but adequate calibration at all patient survival timepoints. In stratified analyses, SB and SOFT scores performed better in younger (< 40 y) and higher Model for End-Stage Liver Disease (≥ 25) patients. All prediction scores had declining discrimination over time, and scores relying on donor factors alone had poor performance. Although the SB and SOFT scores had the best overall performance, all models demonstrated declining performance over time. This underscores the importance of periodically updating and/or developing new prediction models to reflect the evolving transplant field. Scores relying on donor factors alone do not meaningfully inform post-transplant risk.

A Prognostic Nomogram Survival Model for Newly Diagnosed Patients with AIDS-Related Diffuse Large B-Cell Lymphoma: A Multicenter Cohort Study in China

BACKGROUND The prognosis of patients with AIDS-related diffuse large B-cell lymphoma(AR-DLBCL) becomes very poor when the lymphoma relapses or is refractory to first line immunochemotherapy. We aimed to develop a novel ARDPI nomogram prognostic model for risk stratification so as to guide individualized treatment to achieve sustained remission and improve the overall prognosis for newly-diagnosed AR-DLBCL patients. METHODS We interrogated data from 306 patients with newly-diagnosed AR-DLBCL. We filtered variables using LASSO regression and Cox regression to identify prognostic co-variates and develop a survival model, we termed AR-DLBCL Prognostic Index (ARDPI). We evaluated model discrimination, calibration and clinical benefit by Area Under the Receiver-Operator Characteristic (AUROC), calibration plots and decision curve analysis (DCA). Next, we compared the ARDPI model discrimination, calibration and clinical benefit with the IPI and NCCN-IPI models using the same methods. Lastly, we stratified patients into three survival risk cohorts based on ARDPI model by X-tile selecting cutoff point. RESULTS 7 co-variates were independently correlated with survival and were used to develop the ARDPI model including age, lymphocyte monocyte ratio (LMR), CD5 expression on lymphoma cells, blood EBV-DNA copy number, CD4/CD8 ratio, central nervous system (CNS) involvement and anti-HIV therapy (ART). AUROCs of ARDPI model for 1-, 3-, and 5-year were 0.80 (95% Confidence Interval [CI], 0.72, 0.88), 0.78 (0.69, 0.87) and 0.77 (0.63, 0.91). Predicted and calibrated values were concordant. The DCA curve had higher net benefit using the ARDPI model. Prediction accuracy of the ARDPI model was better compared with the IPI and NCCN-IPI models. For example, 3-year survival AUROC in the ARDPI model was 0.78 (0.69, 0.87) compared with the IPI (0.53 [0.43, 0.63] P&amp;lt; 0.001) and the NCCN-IPI (0.52 [0.42, 0.62] P&amp;lt; 0.001). Using the ARDPI model, we identified 3 survival risk cohorts with 3-year survivals of 0.80 , 0.38 and 0.09 ( P&amp;lt;0.001). CONCLUSION The ARDPI has good survival prediction accuracy in newly-diagnosed persons with AIDS-related DLBCL and using it has clinical benefit.Accuracy is better than the IPI and NCCN-IPI models. Validation of our conclusions is needed.

Development of a nomogram for the prediction of acute kidney injury after liver transplantation: a model based on clinical parameters and postoperative cystatin C level

Background Acute kidney injury (AKI) is common after liver transplantation (LT). We developed a nomogram model to predict post-LT AKI. Methods A total of 120 patients were eligible for inclusion in the study. Clinical information was extracted from the institutional electronic medical record system. Blood samples were collected prior to surgery and immediately after surgery. Univariable and multivariate logistic regression were used to identify independent risk factors. Finally, a nomogram was developed based on the final multivariable logistic regression model. Results In total, 58 (48.3%) patients developed AKI. Multivariable logistic regression revealed four independent risk factors for post-LT AKI: operation duration [odds ratio (OR) = 1.728, 95% confidence interval (CI) = 1.121–2.663, p = 0.013], intraoperative hypotension (OR = 3.235, 95% CI = 1.316–7.952, p = 0.011), postoperative cystatin C level (OR = 1.002, 95% CI = 1.001–1.004, p = 0.005) and shock (OR = 4.002, 95% CI = 0.893–17.945, p = 0.070). Receiver operating characteristic curve analysis was used to evaluate model discrimination. The area under the curve value was 0.815 (95% CI = 0.737–0.894). Conclusion The model based on combinations of clinical parameters and postoperative cystatin C levels had a higher predictive performance for post-LT AKI than the model based on clinical parameters or postoperative cystatin C level alone. Additionally, we developed an easy-to-use nomogram based on the final model, which could aid in the early detection of AKI and improve the prognosis of patients after LT.

Convergent Validity of 2 Widely Used Methodologies for Calculating the Hospital Standardized Mortality Ratio in Flanders, Belgium.

To assess their construct validity, we compared results from 2 models used for estimating hospital standardized mortality ratios (HSMRs) in Belgium. The method of the Flemish Hospital Network (FHN) is based on a logistic regression for each of the 64 All Patient Refined Diagnosis-Related Groups that explain 80% of mortality and uses the Elixhauser score to correct for comorbidities. (H)SMRs published on the 3M-Benchmark-Portal are calculated by a simpler indirect standardization for All Patient Refined Diagnosis-Related Groups and risk of mortality (ROM) at discharge. We used administrative data from all eligible hospital admissions in 22 Flemish hospitals between 2016 and 2019 (FHN, n = 682,935; 3M, n = 2,122,305). We evaluated model discrimination and accuracy and assessed agreement in estimated HSMRs between methods. The Spearman correlation between HSMRs generated by the FHN model and the standard 3M model was 0.79. Although 2 of 22 hospitals showed opposite classification results, that is, an HSMR significantly <1 according to the FHN method but significantly >1 according to the 3M model, classification agreement between methods was significant (agreement for 59.1% of hospitals, κ = 0.45). The 3M model (c statistic = 0.96, adjusted Brier score = 26%) outperformed the FHN model (0.87, 17%). However, using ROM at admission instead of at discharge in the 3M model significantly reduced model performance (c statistic = 0.94, adjusted Brier score = 21%), but yielded similar HSMR estimates and eliminated part of the discrepancy with FHN results. Results of both models agreed relatively well, supporting convergent validity. Whereas the FHN method only adjusts for disease severity at admission, the ROM indicator of the 3M model includes diagnoses not present on admission. Although diagnosis codes generated by complications during hospitalization have the tendency to increase the predictive performance of a model, these should not be included in risk adjustment procedures.

Nomogram Analysis Based on Clinical and Sonographic Characteristics for the Assessment of Postpartum Stress Urinary Incontinence.

We aimed to develop and validate a nomogram integrating clinical and sonographic characteristics for the individualized SUI risk evaluation in the early postpartum stage. This was a prospective cross-sectional study. From June 2020 to September 2022, singleton primiparas who underwent TPUS examination at 6-8 weeks postpartum were recruited. They were divided into the training and validation cohorts at a ratio of 8:2 according to the temporal split. All subjects were interviewed before TPUS examination. Univariate and multivariate logistic analyses were performed to develop three models: the clinical, sonographic, and combined models. The ROC curve was plotted to evaluate model discrimination ability. Finally, the combined model was selected to establish the nomogram. The nomogram's discrimination, calibration, and clinical usefulness were evaluated in the training and validation cohorts. The performance of the combined model was better than that of the clinical and sonographic models. Six predictors (BMI, delivery mode, lateral episiotomy, SUI during pregnancy, cystocele, and bladder neck funneling) remained in the combined model. The nomogram based on the combined model had good discrimination with AUCs of 0.848 (95% CI: 0.796-0.900) and 0.872 (95% CI: 0.789-0.955) in the training and validation cohorts, respectively, and the calibration curve showed good efficiency in assessing postpartum SUI. Decision curve analysis showed that the nomogram was clinically useful. The nomogram based on clinical and sonographic characteristics showed good efficiency in assessing postpartum SUI risk and can be a convenient and reliable tool for individual SUI risk assessment.

Nomogram for predicting 90-day mortality in patients with Acinetobacter baumannii-caused hospital-acquired and ventilator-associated pneumonia in the respiratory intensive care unit

We built a prediction model of mortality risk in patients the with Acinetobacter baumannii (AB)-caused hospital-acquired (HAP) and ventilator-associated pneumonia (VAP). In this retrospective study, 164 patients with AB lower respiratory tract infection were admitted to the respiratory intensive care unit (RICU) from January 2019 to August 2021 (29 with HAP, 135 with VAP) and grouped randomly into a training cohort (n = 115) and a validation cohort (n = 49). Least absolute shrinkage and selection operator regression and multivariate Cox regression were used to identify risk factors of 90-day mortality. We built a nomogram prediction model and evaluated model discrimination and calibration using the area under the receiver operating characteristic curve (AUC) and calibration curves, respectively. Four predictors (days in intensive care unit, infection with carbapenem-resistant AB, days of carbapenem use within 90 days of isolating AB, and septic shock) were used to build the nomogram. The AUC of the two groups was 0.922 and 0.823, respectively. The predictive model was well-calibrated; decision curve analysis showed the proposed nomogram would obtain a net benefit with threshold probability between 1% and 100%. The nomogram model showed good performance, making it useful in managing patients with AB-caused HAP and VAP.

Establishment and Verification of a Nomogram for Predicting the Probability of New-Onset Atrial Fibrillation After Dual-Chamber Pacemaker Implantation.

This study aims to establish and validate a nomogram as a predictive model in patients with new-onset atrial fibrillation (AF) after dual-chamber cardiac implantable electronic device (pacemaker) implantation. A total of 1120 Chinese patients with new-onset AF after pacemaker implantation were included in this retrospective study. Patients had AF of at least 180/minute lasting 5 minutes or longer, detected by atrial lead and recorded at least 3 months after implantation. Patients with previous atrial tachyarrhythmias before device implantation were excluded. A total of 276 patients were ultimately enrolled, with 51 patients in the AF group and 225 patients in the non-AF group. Least absolute shrinkage and selection operator (LASSO) method was used to determine the best predictors. Through multivariate logistic regression analysis, a nomogram was drawn as a predictive model. Concordance index, calibration plot, and decision curve analyses were applied to evaluate model discrimination, calibration, and clinical applicability. Internal verification was performed using a bootstrap method. The LASSO method regression analysis found that variables including peripheral arterial disease, atrial pacing-ventricular pacing of at least 50%, atrial sense-ventricular sense of at least 50%, increased left atrium diameter, and age were important predictors of developing AF. In multivariate logistic regression, peripheral arterial disease, atrial pacing-ventricular pacing of at least 50%, and age were found to be independent predictors of new-onset AF. This nomogram may help physicians identify patients at high risk of new-onset AF after pacemaker implantation at an early stage in a Chinese population.

Development and Validation of the HIV-CARDIO-PREDICT Score to Estimate the Risk of Cardiovascular Events in HIV-Infected Patients.

Commonly used risk assessment tools for cardiovascular disease might not be accurate for HIV-infected patients. We aimed to develop a model to accurately predict the 10-year cardiovascular disease (CV) risk of HIV-infected patients. In this retrospective cohort study, adult HIV-infected patients seen at Boston Medical Center between March 2012 and January 2017 were divided into model development and validation cohorts. Boston Medical Center, a tertiary, academic medical center. Adult HIV-infected patients, seen in inpatient and outpatient setting. We used logistic regression to create a prediction risk model for cardiovascular events using data from the development cohort. Using a point-based risk-scoring system, we summarized the relationship between risk factors and cardiovascular disease (CVD) risk. We then used the area under the receiver operating characteristics curve (AUC) to evaluate model discrimination. Finally, we tested the model using a validation cohort. 1914 individuals met the inclusion criteria. The model had excellent discrimination for CVD risk [AUC 0.989; (95% CI: 0.986-0.993)] and included the following 11 variables: male sex (95% CI: 2.53-3.99), African American race/ethnicity (95% CI: 1.50-3.13), current age (95% CI: 0.07-0.13), age at HIV diagnosis (95% CI: -0.10-(-0.02)), peak HIV viral load (95% CI: 9.89 × 10-7-3.00 × 10-6), nadir CD4 lymphocyte count (95% CI: -0.03-(-0.02)), hypertension (95% CI: 0.20-1.54), hyperlipidemia (95% CI: 3.03-4.60), diabetes (95% CI: 0.61-1.89), chronic kidney disease (95% CI: 1.26-2.62), and smoking (95% CI: 0.12-2.39). The eleven-parameter multiple logistic regression model had excellent discrimination [AUC 0.957; (95% CI: 0.938-0.975)] when applied to the validation cohort. Our novel HIV-CARDIO-PREDICT Score may provide a rapid and accurate evaluation of CV disease risk among HIV-infected patients and inform prevention measures.

Preoperative Factors Predict Memory Decline After Coronary Artery Bypass Grafting or Percutaneous Coronary Intervention in an Epidemiological Cohort of Older Adults.

Background Durable memory decline may occur in older adults after surgical (coronary artery bypass grafting [CABG]) or nonsurgical (percutaneous coronary intervention) coronary revascularization. However, it is unknown whether individual memory risk can be predicted. We reanalyzed an epidemiological cohort of older adults to predict memory decline at ≈1 year after revascularization. Methods and Results We studied Health and Retirement Study participants who underwent CABG or percutaneous coronary intervention at age ≥65 years between 1998 and 2015 and participated in ≥1 biennial postprocedure assessment. Using a memory score based on direct and proxy cognitive tests, we identified participants whose actual postprocedure memory score was 1-2 ("mild") or >2 ("major") SDs below expected postprocedure performance. We modeled probability of memory decline using logistic regression on preoperatively known factors and evaluated model discrimination and calibration. A total of 1390 participants (551 CABG, 839 percutaneous coronary intervention) underwent CABG/percutaneous coronary intervention at 75±6 years old; 40% were women. The cohort was 83% non-Hispanic White, 8.4% non-Hispanic Black, 6.4% Hispanic ethnicity, and 1.7% from other groups masked by the HRS (Health and Retirement Study) to preserve participant confidentiality. At a median of 1.1 (interquartile range, 0.6-1.6) years after procedure, 267 (19%) had mild memory decline and 88 (6.3%) had major memory decline. Factors predicting memory decline included older age, frailty, and off-pump CABG; obesity was protective. The optimism-corrected area under the receiver operator characteristic curve was 0.73 (95% CI, 0.71-0.77). A cutoff of 50% probability of memory decline identified 14% of the cohort as high risk, and was 94% specific and 30% sensitive for late memory decline. Conclusions Preoperative factors can be used to predict late memory decline after coronary revascularization in an epidemiological cohort with high specificity.

Available Risk Scores Modestly Predict Hemorrhage in Patients with Atrial Fibrillation and Cance

Background: Patients with cancer have an increased risk of developing atrial fibrillation (AF). Patients with cancer who develop AF have an increased risk of cardiovascular-related death at 12-months. Use of anticoagulant therapy may benefit these patients. However, patients with cancer are at increased risk of anticoagulant-related bleeding. Several risk prediction models are available to estimate the risk of anticoagulant-related bleeding in patients with AF. However, these models were developed predominately in patients without cancer. Thus, the utility of these models in cancer is unknown. We aimed to evaluate the performance of existing bleeding risk models in a cohort of patients with active cancer and AF starting on anticoagulant therapy. Methods: Using a nationwide cohort of US Veterans, we identified patients with active cancer and AF with a new prescription for anticoagulant therapy between 2012 and 2018. Cancer was defined as active if the diagnosis of AF occurred within 3 months prior to up to 6 months after new cancer diagnosis or in the setting of incurable hematological cancers (e.g., multiple myeloma, chronic myelogenous leukemia) or metastatic solid tumors. We identified anticoagulant-related major bleeding (MB) within 6 months of anticoagulant start by the presence of inpatient ICD-9/10 codes in any position as previously validated. Patients were censored 30-days after last anticoagulant prescription. We evaluated the predictive performance of six models. The association per standard deviation increase in the score and MB within 6 months of anticoagulant start was estimated using Fine-Gray subdistribution survival analysis to account for the competing risk of death≥÷A sensitivity analysis limiting inpatient ICD-9/10 codes for MB to the first two positions was also performed. We evaluated model discrimination using Harrell's c-statistic. Results: 5716 patients met inclusion criteria into the cohort. The most common types of cancer included: hematological malignancies (24%), lung (20%), prostate (18%), non-prostate genitourinary (11%), and gastrointestinal (10%) cancer. Mean age at the start of anticoagulant therapy was 73.4 years and median overall survival 41.1 months. Of the cohort, anticoagulant therapy was as follows: 2040 warfarin, 179 LMWH, 5 fondaparinux, and 3495 DOAC. 328 (5.7%) patients had a MB within 6 months of anticoagulant start and 439 (7.7%) within 12 months. The most common type of bleeds within the first 6 months of anticoagulant therapy was gastrointestinal (47%), followed by hematuria (28%) and hemoptysis (13.7%). 2.4% of bleeding events (n=8) were intracranial hemorrhage. The increase in hazard of bleeding per each standard deviation increase in bleeding score is in Table 1. For model discrimination, Harrell's c-statistic for each score is also listed in Table 1. Conclusions: In this cohort of 5716 patients with active cancer and atrial fibrillation starting on anticoagulant therapy, available clinical risk models moderately predicted bleeding. The addition of cancer-specific risk factors might improve accuracy of the available models. Figure 1View largeDownload PPTFigure 1View largeDownload PPT Close modal

Abstract 13653: Leveraging Natural Language Processing and Machine Learning to Predict Worsening Heart Failure Events

Background: Prior risk models in patients with heart failure (HF) have focused on hospitalizations for worsening HF (WHF) and have not evaluated for differences in predictors by left ventricular ejection fraction (LVEF). We used natural language processing (NLP) and machine learning methods with access to longitudinal electronic health record (EHR) data to develop risk prediction models for WHF events across practice settings and by LVEF category. Methods: We identified all adults with HF and known LVEF on January 1 st of each year from 2011-2019 in an integrated health care system. WHF events within 1 year were defined as any hospitalization, emergency department, or outpatient encounter with ≥1 symptom, ≥2 objective findings including ≥1 sign, and ≥1 change in HF-related therapy. Signs and symptoms were ascertained using rule-based NLP. We conducted boosted decision tree-based ensemble models for any WHF event within each LVEF category: HF with reduced EF (HFrEF; LVEF ≤40%), HF with mildly reduced EF (HFmrEF; LVEF 41-49%), and HF with preserved EF (HFpEF; LVEF ≥50%). We evaluated model discrimination using area under the curve (AUC) and model calibration using Brier scores. Results: Among 359,298 patients from 2011-2019, 65,838 (18%) had HFrEF, 52,491 (15%) had HFmrEF, and 240,969 (67%) had HFpEF. Mean age was 75±12, 47% were women, and 37% were minorities including 10% Black, 11% Asian/Pacific Islander, and 12% of Hispanic ethnicity. WHF events occurred in 22% of patients with HFrEF, 17% with HFmrEF, and 16% with HFpEF. The models displayed an AUC of 0.75 and Brier score of 0.15 for HFrEF and an AUC of 0.77 and Brier scores of 0.12 for both HFmrEF and HFpEF. Clinical predictors were similar across LVEF categories ( Table ). Conclusions: Longitudinal EHR data can be leveraged using NLP and machine learning for accurate risk estimation that reliably identifies clinical predictors across a range of LVEF. These findings may provide novel insight into the natural history of HF.

Evaluating and Optimizing Prognostic Modeling for Patients Undergoing Stereotactic Body Radiotherapy for Bone Metastases

<h3>Purpose/Objective(s)</h3> Stereotactic body radiotherapy (SBRT) holds an expanding role in the treatment of bone metastases. As compared to conventional external beam radiation (cEBRT), SBRT may be best applied to specific subsets of patients with more favorable prognosis and may prolong survival time. This calls into question the generalizability of survival estimates from prognostic models built from more general patient populations treated with cEBRT. We investigate the external validity of three such models for patients treated with bone-directed SBRT and identify approaches to optimize prognostication for this population. <h3>Materials/Methods</h3> Patients treated with SBRT for bone metastases from 1/2017-12/2019 at 3 radiation oncology facilities were identified through retrospective electronic medical record review. Demographic, disease, and treatment data were collected, including the covariates of 3 previously-validated models built in populations that excluded SBRT: the 27-item Bone Metastases Ensemble Trees for Survival (BMETS) machine-learning model and 2 Cox regression models—Chow's 3-item (C-3) and Westhoff's 2-item (W-2) tools. Individualized estimates for risk of death at various time points after consultation were obtained using each model. Time-dependent area under the curve (tAUC), calibration plots, and Brier scores evaluated model discrimination, calibration, and overall performance when each model was applied to the SBRT validation set. <h3>Results</h3> A total of 192 sites in 168 patients were included. Patients were 65.5% male; 74% white, 17% black and 8% other; with median age of 63 years (range: 18-92), and median Karnofsky Performance Status of 90 (range: 50-100). Histologies were prostate (31%), lung (15%), breast (10%), and other (44%). Estimated median overall survival was 3.3 years. BMETS tAUC was 0.73, 0.81, and 0.81 at 6, 9, and 12 months, with significant outperformance (p<0.05) of C-3 and W-2 models by the 12-month time point (tAUC 0.71 and 0.75, respectively). Calibration plots showed overestimation of risk of death in all 3 models. Brier scores for BMETS, C-3 and W-2 were respectively 0.24, 0.28 and 0.25 at 12 months; the simple baseline model from the Kaplan-Meier curve estimate (Brier score 0.19) outperformed all 3 models. Subsequent analyses identified variables absent from the 3 models strongly predictive of survival in the SBRT cohort, including number of metastatic lesions. <h3>Conclusion</h3> When applied to an SBRT cohort, BMETS maintained excellent discrimination at time points >6 months, outperforming C-2 and W-3 models. Yet, each model underestimated specific survival probabilities, suggesting that tools built from non-SBRT populations may be poorly calibrated for SBRT and supporting use of SBRT-specific prognostic tools. To address this, we have modified the BMETS web platform to permit for users to select for SBRT and non-SBRT specific survival predictions.

Development and validation of an early warning model for hospitalized COVID-19 patients: a multi-center retrospective cohort study

BackgroundTimely identification of deteriorating COVID-19 patients is needed to guide changes in clinical management and admission to intensive care units (ICUs). There is significant concern that widely used Early warning scores (EWSs) underestimate illness severity in COVID-19 patients and therefore, we developed an early warning model specifically for COVID-19 patients.MethodsWe retrospectively collected electronic medical record data to extract predictors and used these to fit a random forest model. To simulate the situation in which the model would have been developed after the first and implemented during the second COVID-19 ‘wave’ in the Netherlands, we performed a temporal validation by splitting all included patients into groups admitted before and after August 1, 2020. Furthermore, we propose a method for dynamic model updating to retain model performance over time. We evaluated model discrimination and calibration, performed a decision curve analysis, and quantified the importance of predictors using SHapley Additive exPlanations values.ResultsWe included 3514 COVID-19 patient admissions from six Dutch hospitals between February 2020 and May 2021, and included a total of 18 predictors for model fitting. The model showed a higher discriminative performance in terms of partial area under the receiver operating characteristic curve (0.82 [0.80–0.84]) compared to the National early warning score (0.72 [0.69–0.74]) and the Modified early warning score (0.67 [0.65–0.69]), a greater net benefit over a range of clinically relevant model thresholds, and relatively good calibration (intercept = 0.03 [− 0.09 to 0.14], slope = 0.79 [0.73–0.86]).ConclusionsThis study shows the potential benefit of moving from early warning models for the general inpatient population to models for specific patient groups. Further (independent) validation of the model is needed.

Prediction of Sudden Cardiac Death Manifesting With Documented Ventricular Fibrillation or Pulseless Ventricular Tachycardia.

This study aimed to develop a novel clinical prediction algorithm for avertable sudden cardiac death. Sudden cardiac death manifests as ventricular fibrillation (VF)/ ventricular tachycardia (VT) potentially treatable with defibrillation, or nonshockable rhythms (pulseless electrical activity/asystole) with low likelihood of survival. There are no available clinical risk scores for targeted prediction of VF/VT. Subjects with out-of-hospital sudden cardiac arrest presenting with documented VF or pulseless VT (33% of total cases) were ascertained prospectively from the Portland, Oregon, metro area with population ≈1 million residents (n=1,374, 2002-2019). Comparisons of lifetime clinical records were conducted with a control group (n=1,600) with ≈70% coronary disease prevalence. Prediction models were constructed from a training dataset using backwards stepwise logistic regression and applied to an internal validation dataset. Receiver operating characteristic curves (Cstatistic) were used to evaluate model discrimination. External validation was performed in a separate, geographically distinct population (Ventura County, California, population ≈850,000, 2015-2020). A clinical algorithm (VFRisk) constructed with 13 clinical, electrocardiogram, and echocardiographic variableshad very good discrimination in the training dataset (C statistic=0.808; [95% CI: 0.774-0.842]) and was successfully validated in internal (C statistic=0.776 [95% CI: 0.725-0.827]) and external (C statistic=0.782 [95% CI: 0.718-0.846]) datasets. The algorithm substantially outperformed the left ventricular ejection fraction (LVEF)≤35% (Cstatistic=0.638) and performed well across the LVEF spectrum. An algorithm for prediction of sudden cardiac arrest manifesting with VF/VT was successfully constructed using widely available clinical and noninvasive markers. These findings have potential to enhance primaryprevention, especially in patients with mid-range or preserved LVEF.

Development of Deep Learning Models for Predicting In-Hospital Mortality Using an Administrative Claims Database: Retrospective Cohort Study.

BackgroundAdministrative claims databases have been used widely in studies because they have large sample sizes and are easily available. However, studies using administrative databases lack information on disease severity, so a risk adjustment method needs to be developed.ObjectiveWe aimed to develop and validate deep learning–based prediction models for in-hospital mortality of acute care patients.MethodsThe main model was developed using only administrative claims data (age, sex, diagnoses, and procedures on the day of admission). We also constructed disease-specific models for acute myocardial infarction, heart failure, stroke, and pneumonia using common severity indices for these diseases. Using the Japanese Diagnosis Procedure Combination data from July 2010 to March 2017, we identified 46,665,933 inpatients and divided them into derivation and validation cohorts in a ratio of 95:5. The main model was developed using a 9-layer deep neural network with 4 hidden dense layers that had 1000 nodes and were fully connected to adjacent layers. We evaluated model discrimination ability by an area under the receiver operating characteristic curve (AUC) and calibration ability by calibration plot.ResultsAmong the eligible patients, 2,005,035 (4.3%) died. Discrimination and calibration of the models were satisfactory. The AUC of the main model in the validation cohort was 0.954 (95% CI 0.954-0.955). The main model had higher discrimination ability than the disease-specific models.ConclusionsOur deep learning–based model using diagnoses and procedures produced valid predictions of in-hospital mortality.

Abstract 14426: Comparing Performance of MARKER-HF, a Machine Learning Derived Risk Score, and the Seattle Heart Failure Model in Patients With Heart Failure in a Large, Integrated Health System

Introduction: Patients with heart failure (HF) who advance from Stage C to Stage D experience progressively higher morbidity and mortality rates. Cardiovascular societies recommend using 1-year mortality risk (≥20%) as a trigger for evaluation by a HF specialist. Application of available risk scores to identify patients at increased risk using electronic health records (EHR) is challenging. Consequently, we compared the performance of MARKER-HF, a machine-learning risk model that uses readily available structured data in the EHR, to the Seattle Heart Failure Model (SHFM), which requires elements derived from both structured and unstructured data, for estimating 1-year mortality risk. Methods: We analyzed an EHR database for ambulatory patients with HF seen in primary care and cardiology clinics from 1/1/10-12/31/19 at Northwestern Medicine. Eligibility criteria included ages ≥18 years old, at least 1-year of follow-up or death within 1-year, no history of heart transplantation or left ventricular assist device implantation, and availability of the 8 variables used to calculate MARKER-HF (diastolic blood pressure, creatinine, blood urea nitrogen, hemoglobin, white blood cell count, platelets, albumin, and red blood cell distribution width). Unavailable or missing variables required for SHFM were imputed. In all HF patients and in those with reduced ejection fraction (HFrEF), we evaluated model discrimination for 1-year mortality using area under the curve (AUC) and compared AUCs with the DeLong test. Results: In a sample of 7130 patients, of whom 1012 died within 1-year, 46.9% were female and median age was 71 (IQR 60-82). The AUC for MARKER-HF and SHFM were 0.69 and 0.64, respectively, for all HF patients and 0.69 and 0.67, respectively, for patients with HFrEF (n=1899). The AUCs for MARKER-HF and SHFM were different for the entire HF population (DeLong p-value=0.01) but not in the HFrEF sub-group. Conclusion: In patients with ambulatory HF at an integrated health system, MARKER-HF produced 1-year mortality estimates with equal (HFrEF) or higher (all HF) discrimination than SFHM using fewer and more readily available variables. These findings suggest that MARKER-HF may facilitate the identification of patients with advanced HF using EHR data.