Poor Calibration Research Articles

External Validation of the FIPS Score for Post-TIPS Mortality in a National Veterans Affairs Cohort.

The Freiburg index of post-TIPS survival (FIPS) score was recently demonstrated to improve prediction of post-TIPS mortality relative to existing standards. As this score was derived from a German cohort over an extended time period, it is unclear if performance will translate well to other settings. This study aimed to externally validate the FIPS score in a large Veterans Affairs (VA) cohort over two separate eras of TIPS-related care. This was a retrospective cohort study of patients with cirrhosis who underwent TIPS placement in the VA from 2008 to 2020.Cox regression models for post-TIPS survival were constructed using FIPS, MELD, MELD-Na, or CTP scores as predictors. Discrimination (Harrell's C) and calibration (joint tests of calibration curve slope and intercept) were evaluated for each score. A stratified analysis was performed for time periods between 2008-2013 and 2014-2020. The cohort of 1,274 patients was 97.3% male with mean age 60.9years and mean MELD-Na 14.The FIPS score demonstrated the highest overall discrimination versus MELD, MELD-Na, and CTP (0.634 vs. 0.585, 0.626, 0.612, respectively). However, in the modern treatment era (2014-2020), the FIPS score performed similarly to MELD-Na. Additionally, the FIPS score demonstrated poor calibration at one-month and six-month post-TIPS timepoints (joint p = 0.04 and 0.004, respectively). MELD, MELD-Na, and CTP were well-calibrated at each timepoint (each joint p > 0.05). The FIPS score performed similarly to MELD-Na in the modern TIPS treatment era and demonstrated regions of poor calibration. Future models derived with contemporary data may improve prediction of post-TIPS mortality.

Hypothermia Outcome Prediction after Extracorporeal Life Support for Hypothermic Cardiac Arrest Patients: Assessing the Performance of the HOPE Score in Case Reports from the Literature.

Aims: The hypothermia outcome prediction after extracorporeal life support (ECLS) score, or HOPE score, provides an estimate of the survival probability in hypothermic cardiac arrest patients undergoing ECLS rewarming. The aim of this study was to assess the performance of the HOPE score in case reports from the literature. Methods: Cases were identified through a systematic review of the literature. We included cases of hypothermic cardiac arrest patients rewarmed with ECLS and not included in the HOPE derivation and validation studies. We calculated the survival probability of each patient according to the HOPE score. Results: A total of 70 patients were included. Most of them (62/70 = 89%) survived. The discrimination using the HOPE score was good (Area Under the Receiver Operating Characteristic Curve = 0.78). The calibration was poor, with HOPE survival probabilities averaging 54%. Using a HOPE survival probability threshold of at least 10% as a decision criterion for rewarming a patient would have resulted in only five false positives and a single false negative, i.e., 64 (or 91%) correct decisions. Conclusions: In this highly selected sample, the HOPE score still had a good practical performance. The selection bias most likely explains the poor calibration found in the present study, with survivors being more often described in the literature than non-survivors. Our finding underscores the importance of working with a representative sample of patients when deriving and validating a score, as was the case in the HOPE studies that included only consecutive patients in order to minimize the risk of publication bias and lower the risk of overly optimistic outcomes.

Probabilistic Forecasts of Arctic Sea Ice Thickness

In recent decades, warming temperatures have caused sharp reductions in the volume of sea ice in the Arctic Ocean. Predicting changes in Arctic sea ice thickness is vital in a changing Arctic for making decisions about shipping and resource management in the region. We propose a statistical spatio-temporal two-stage model for sea ice thickness and use it to generate probabilistic forecasts up to three months into the future. Our approach combines a contour model to predict the ice-covered region with a Gaussian random field to model ice thickness conditional on the ice-covered region. Using the most complete estimates of sea ice thickness currently available, we apply our method to forecast Arctic sea ice thickness. Point predictions and prediction intervals from our model offer comparable accuracy and improved calibration compared with existing forecasts. We show that existing forecasts produced by ensembles of deterministic dynamic models can have large errors and poor calibration. We also show that our statistical model can generate good forecasts of aggregate quantities such as overall and regional sea ice volume. Supplementary materials accompanying this paper appear on-line.

Validation and public health modelling of risk prediction models for kidney cancer using the UK Biobank.

To externally validate risk models for the detection of kidney cancer, as early detection of kidney cancer improves survival and stratifying the population using risk models could enable an individually tailored screening programme. We validated the performance of 30 existing phenotypic models predicting the risk of kidney cancer in the UK Biobank cohort (n = 450 687). We compared the discrimination and calibration of models for men, women, and a mixed-sex cohort. Population level data were used to estimate model performance in a screening scenario for a range of risk thresholds (6-year risk: 0.1-1.0%). In all, 10 models had reasonable discrimination (area under the receiver-operating characteristic curve >0.60), although some had poor calibration. Modelling demonstrated similar performance of the best models over a range of thresholds. The models showed an improvement in ability to identify cases compared to age- and sex-based screening. All the models performed less well in women than men. The present study is the first comprehensive external validation of risk models for kidney cancer. The best-performing models are better at identifying individuals at high risk of kidney cancer than age and sex alone; however, the benefits are relatively small. Feasibility studies are required to determine applicability to a screening programme.

An Innovative Sensor Fusion Algorithm for Motion Tracking With On-Line Bias Compensation: Application to Joint Angles Estimation in Yoga

Magnetic Inertial Measurement Units (MIMUs) represent an increasingly used technology in the field of human motion. However, their applications are strongly affected by magnetic disturbances and poor calibration, which could determine inaccurate attitude estimations. Thus, Inertial Measurement Units (IMUs), relying exclusively on accelerometers and gyroscopes, could be used as an alternative to MIMUs for motion tracking, avoiding the negative effect of magnetic disturbances. Unfortunately, gyroscope signals are characterized by several error sources, among which gyroscope bias, whose variations due to environmental factors and temporal instability strongly affect the attitude estimation accuracy. Aim of the present work is to propose a novel sensor fusion algorithm for IMU-based applications that embodies an adaptive on-line bias capture module. The accuracy of the proposed filter was tested on ten expert yoga practitioners during the execution of a sun salutation sequence. The right upper limb joint angles were estimated and their accuracy assessed in terms of Mean Absolute Error (MAE) and Pearson’s correlation coefficient by comparison with an optoelectronic motion capture system. The achieved worst-case accuracy was 5.59, 6.75 and 3.49 degrees for the wrist, elbow and shoulder joints respectively. The accuracy of the algorithm is further confirmed by the high values of the Pearson’s correlation coefficients between IMUs and OMC angles, which were greater than 0.91, 0.92 and 0.98 respectively for the three considered joints. The proposed algorithm can thus be considered as a promising tool for attitude estimation in those contexts when magnetic interferences cannot be monitored and, thus, adequately compensated.

Can the adverse childhood experiences (ACEs) checklist be utilized to predict emergency department visits among children and adolescents?

BackgroundExtensive literature has shown an association of Adverse Childhood Experiences (ACEs) with adverse health outcomes; however, its ability to predict events or stratify risks is less known. Individuals with mental illness and ACE exposure have been shown to visit emergency departments (ED) more often than those in the general population. This study thus examined the ability of the ACEs checklist to predict ED visits within the subsequent year among children and adolescents presenting to mental health clinics with pre-existing mental health issues.MethodsThe study analyzed linked data (n = 6100) from two databases provided by Alberta Health Services (AHS). The Regional Access and Intake System (RAIS 2016–2018) database provided data on the predictors (ACE items, age, sex, residence, mental health program type, and primary diagnosis) regarding children and adolescents (aged 0–17 years) accessing addiction and mental health services within Calgary Zone, and the National Ambulatory Care Reporting System (NACRS 2016–2019) database provided data on ED visits. A 25% random sample of the data was reserved for validation purposes. Two Least Absolute Shrinkage and Selection Operator (LASSO) logistic regression models, each employing a different method to tune the shrinkage parameter lambda (namely cross-validated and adaptive) and performing 10-fold cross-validation for a set of 100 lambdas in each model were examined.ResultsThe adaptive LASSO model had a slightly better fit in the validation dataset than the cross-validated model; however, it still demonstrated poor discrimination (AUC 0.60, sensitivity 37.8%, PPV 49.6%) and poor calibration (over-triaged in low-risk and under-triaged in high-risk subgroups). The model’s poor performance was evident from an out-of-sample deviance ratio of − 0.044.ConclusionThe ACEs checklist did not perform well in predicting ED visits among children and adolescents with existing mental health concerns. The diverse causes of ED visits may have hindered accurate predictions, requiring more advanced statistical procedures. Future studies exploring other machine learning approaches and including a more extensive set of childhood adversities and other important predictors may produce better predictions. Furthermore, despite highly significant associations being observed, ACEs may not be deterministic in predicting health-related events at the individual level, such as general ED use.

Estimating Bleeding Risk in Patients with Cancer-Associated Thrombosis: Evaluation of Existing Risk Scores and Development of a New Risk Score.

Bleeding risk is highly relevant for treatment decisions in cancer-associated thrombosis (CAT). Several risk scores exist, but have never been validated in patients with CAT and are not recommended for practice. To compare methods of estimating clinically relevant (major and clinically relevant nonmajor) bleeding risk in patients with CAT: (1) existing risk scores for bleeding in venous thromboembolism, (2) pragmatic classification based on cancer type, and (3) new prediction model. In a posthoc analysis of the Hokusai VTE Cancer study, a randomized trial comparing edoxaban with dalteparin for treatment of CAT, seven bleeding risk scores were externally validated (ACCP-VTE, HAS-BLED, Hokusai, Kuijer, Martinez, RIETE, and VTE-BLEED). The predictive performance of these scores was compared with a pragmatic classification based on cancer type (gastrointestinal; genitourinary; other) and a newly derived competing risk-adjusted prediction model based on clinical predictors for clinically relevant bleeding within 6 months after CAT diagnosis with nonbleeding-related mortality as the competing event ("CAT-BLEED"). Data of 1,046 patients (149 events) were analyzed. Predictive performance of existing risk scores was poor to moderate (C-statistics: 0.50-0.57; poor calibration). Internal validation of the pragmatic classification and "CAT-BLEED" showed moderate performance (respective C-statistics: 0.61; 95% confidence interval [CI]: 0.56-0.66, and 0.63; 95% CI 0.58-0.68; good calibration). Existing risk scores for bleeding perform poorly after CAT. Pragmatic classification based on cancer type provides marginally better estimates of clinically relevant bleeding risk. Further improvement may be achieved with "CAT-BLEED," but this requires external validation in practice-based settings and with other DOACs and its clinical usefulness is yet to be demonstrated.

Prediction of Critical Care Outcome for Adult Patients Presenting to Emergency Department Using Initial Triage Information: An XGBoost Algorithm Analysis.

BackgroundThe emergency department (ED) triage system to classify and prioritize patients from high risk to less urgent continues to be a challenge.ObjectiveThis study, comprising 80,433 patients, aims to develop a machine learning algorithm prediction model of critical care outcomes for adult patients using information collected during ED triage and compare the performance with that of the baseline model using the Korean Triage and Acuity Scale (KTAS).MethodsTo predict the need for critical care, we used 13 predictors from triage information: age, gender, mode of ED arrival, the time interval between onset and ED arrival, reason of ED visit, chief complaints, systolic blood pressure, diastolic blood pressure, pulse rate, respiratory rate, body temperature, oxygen saturation, and level of consciousness. The baseline model with KTAS was developed using logistic regression, and the machine learning model with 13 variables was generated using extreme gradient boosting (XGB) and deep neural network (DNN) algorithms. The discrimination was measured by the area under the receiver operating characteristic (AUROC) curve. The ability of calibration with Hosmer–Lemeshow test and reclassification with net reclassification index were evaluated. The calibration plot and partial dependence plot were used in the analysis.ResultsThe AUROC of the model with the full set of variables (0.833-0.861) was better than that of the baseline model (0.796). The XGB model of AUROC 0.861 (95% CI 0.848-0.874) showed a higher discriminative performance than the DNN model of 0.833 (95% CI 0.819-0.848). The XGB and DNN models proved better reclassification than the baseline model with a positive net reclassification index. The XGB models were well-calibrated (Hosmer-Lemeshow test; P>.05); however, the DNN showed poor calibration power (Hosmer-Lemeshow test; P<.001). We further interpreted the nonlinear association between variables and critical care prediction.ConclusionsOur study demonstrated that the performance of the XGB model using initial information at ED triage for predicting patients in need of critical care outperformed the conventional model with KTAS.

Balloon angioplasty: A promising adjunct to arteriovenous fistula creation compared with hydrostatic dilatation in small-caliber cephalic veins.

To maintain the patency and longevity of arteriovenous fistula, the availability of a venous segment with adequate diameter is important. In Indian population, many chronic kidney disease patients have poor caliber veins. The study aimed to evaluate the efficacy of hydrostatic dilatation versus Primary balloon angioplasty of small caliber cephalic veins of (≤2.5 mm) preoperatively in terms of patency rate and maturation time of arteriovenous fistula. Patients (n = 80) with an end-stage renal disease requiring arteriovenous access surgery for hemodialysis with small caliber cephalic veins were randomized into two groups, i.e., hydrostatic dilatation and primary balloon angioplasty, each with 40 patients. All patients underwent a thorough clinical examination as well as duplex ultrasound vein mapping of both upper extremities. Patients were followed up for six months and primary patency, maturation time, and complications were noted. Immediate technical success with good palpable thrill was achieved in 97.5% of patients in the primary balloon angioplasty group and 87.5% in the hydrostatic dilatation group. The fistula maturation time in the primary balloon angioplasty group was 34.41 days and 46.18 days in the hydrostatic dilatation group. In the primary balloon angioplasty group, the primary patency of the fistula was 97.5% and 87.5% in the hydrostatic dilatation group, at six months. The arteriovenous fistula functioning rate was 77.5% in the hydrostatic dilatation group as compared to 92.5% in the primary balloon angioplasty group at six months. The incidence of surgical site infection was 5% in the primary balloon angioplasty group as compared to 10% in the hydrostatic dilatation group. Primary balloon angioplasty of small caliber cephalic veins (≤2.5 mm) performed prior to arteriovenous fistula creation for hemodialysis is a beneficial procedure.

MP45-01 VALIDATION AND PUBLIC HEALTH MODELLING OF RISK PREDICTION MODELS FOR KIDNEY CANCER IN UK BIOBANK

You have accessJournal of UrologyKidney Cancer: Epidemiology & Evaluation/Staging/Surveillance I (MP45)1 Sep 2021MP45-01 VALIDATION AND PUBLIC HEALTH MODELLING OF RISK PREDICTION MODELS FOR KIDNEY CANCER IN UK BIOBANK Hannah Harrison, Lisa Pennells, Angela Wood, Sabrina Rossi, Grant Stewart, Simon Griffin, and Juliet Usher-Smith Hannah HarrisonHannah Harrison More articles by this author , Lisa PennellsLisa Pennells More articles by this author , Angela WoodAngela Wood More articles by this author , Sabrina RossiSabrina Rossi More articles by this author , Grant StewartGrant Stewart More articles by this author , Simon GriffinSimon Griffin More articles by this author , and Juliet Usher-SmithJuliet Usher-Smith More articles by this author View All Author Informationhttps://doi.org/10.1097/JU.0000000000002066.01AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Kidney cancer is the 14th most common cancer worldwide. Although early detection is associated with improved survival rates, many newly diagnosed kidney cancers are metastatic. A barrier to the introduction of a screening programme is low population prevalence. Population risk stratification could minimise harms to individuals and improve the efficiency of a screening programme. Stratification requires a model that accurately identifies individuals at high risk of kidney cancer. Although several models have been developed most have not been externally validated, and the benefits of incorporating them in a screening programme have not been assessed. METHODS: We identified phenotypic risk models in a recent systematic review and validated them in a large population cohort (UK Biobank). We assessed discrimination and calibration of the models for men, women and the whole cohort. We undertook a public health modelling analysis using the best performing models to estimate accuracy in the UK population (individuals aged 40-70). We accounted for differences in demographics (age and sex) and kidney cancer incidence between the UK Biobank cohort and the general population. We compared the ability of the models to identify high-risk individuals for screening with age- and sex-based screening strategies. RESULTS: We included 30 models in the review. Eight had reasonable discrimination (AUROC>0.62) in men, women and the mixed-sex cohort. Many of the models had poor calibration in the UK Biobank. Modelling demonstrated the accuracy of the best models over a range of thresholds (6-year risk: 0.1%-1.0%). At any threshold, the models performed similarly. Overall, they showed a small improvement in ability to identify high-risk individuals compared to age- and sex- based screening. At a cut-off threshold of 0.4%, the best performing model screens 12.3% of the population and detects one case for every 180 individuals screened. Screening all men over the age of 60 (14.1%) would detect one case for every 206 individuals screened. All of the models performed less well in women than men. CONCLUSIONS: This is the first comprehensive external validation of risk prediction models for kidney cancer. Five models showed both reasonable discrimination and good calibration in a UK population. The best-performing models could improve the efficiency of screening by similar amounts, with the choice of model depending on the availability of data. Future research may consider adding biomarkers or genetic risk factors to models. Source of Funding: HH is supported by a NIHR DSE Award (NIHR301182) © 2021 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 206Issue Supplement 3September 2021Page: e802-e802 Advertisement Copyright & Permissions© 2021 by American Urological Association Education and Research, Inc.MetricsAuthor Information Hannah Harrison More articles by this author Lisa Pennells More articles by this author Angela Wood More articles by this author Sabrina Rossi More articles by this author Grant Stewart More articles by this author Simon Griffin More articles by this author Juliet Usher-Smith More articles by this author Expand All Advertisement Loading ...

Comparative performance of the two pooled cohort equations for predicting atherosclerotic cardiovascular disease

Background and aimsMultivariable algorithms have been developed to predict the risk of atherosclerotic cardiovascular disease (ASCVD) to identify high-risk patients. Shortly after the introduction of the AHA/ACC Pooled Cohort Equations (PCE), a systematic overestimation of risk was identified. As such, a revised PCE was proposed to more accurately assess ASCVD risk. This study aims to compare the accuracy of both PCE in predicting ASCVD risk within a large, real-world patient sample in the US. MethodsThis retrospective cohort study identified 20,843 patients aged between 40 and 75 years with no previous ASCVD in an academic healthcare system. Model fit, calibration, and discrimination were compared between PCE using Bayesian Information Criterion (BIC), Hosmer-Lemeshow test, area under the ROC curves (AUC), Brier score, and precision-recall analysis. In addition, we examined race and sex subgroups for effect modification. ResultsBoth PCE showed poor calibration (Hosmer-Lemeshow χ2 > 20; p < 0.05) and discrimination (AUC<0.7). The lack of improvement in discrimination of the revised PCE (AUC: 0.677 vs 0.679; p = 0.357) was confirmed with the AUC precision-recall curves (AUCPR: 0.0717 vs 0.0698). In contrast, the AHA/ACC PCE showed a strong positive risk prediction (ΔBIC>10) compared to the revised PCE, although calibration curves had overlapped. ConclusionsIn this single center analysis, both PCE had poor calibration and discrimination of ASCVD risk in a large, real-world patient sample followed up for over 2 years. There was no evidence of improvement in the accuracy of the revised PCE in assessing the risk of ASCVD in relation to the AHA/ACC PCE.

Use of Time-to-Event Analysis to Develop On-Scene Return of Spontaneous Circulation Prediction for Out-of-Hospital Cardiac Arrest Patients

We aimed to train and validate the time to on-scene return of spontaneous circulation prediction models using time-to-event analysis among out-of-hospital cardiac arrest patients. Using a Korean population-based out-of-hospital cardiac arrest registry, we selected a total of 105,215 adults with presumed cardiac etiologies between 2013 and 2018. Patients from 2013 to 2017 and from 2018 were analyzed for training and test, respectively. We developed 4 time-to-event analyzing models (Cox proportional hazard [Cox], random survival forest, extreme gradient boosting survival, and DeepHit) and 4 classification models (logistic regression, random forest, extreme gradient boosting, and feedforward neural network). Patient characteristics and Utstein elements collected at the scene were used as predictors. Discrimination and calibration were evaluated by Harrell's C-index and integrated Brier score. Among the 105,215 patients (mean age 70 years and 64% men), 86,314 and 18,901 patients belonged to the training and test sets, respectively. On-scene return of spontaneous circulation was achieved in 5,240 (6.1%) patients in the former set and 1,709 (9.0%) patients in the latter. The proportion of emergency medical services (EMS) management was higher and scene time interval longer in the latter. Median time from EMS scene arrival to on-scene return of spontaneous circulation was 8 minutes for both datasets. Classification models showed similar discrimination and poor calibration power compared to survival models; Cox showed high discrimination with the best calibration (C-index [95% confidence interval]: 0.873 [0.865 to 0.882]; integrated Brier score at 30 minutes: 0.060). Incorporating time-to-event analysis could lead to improved performance in prediction models and contribute to personalized field EMS resuscitation decisions.

Accuracy of risk models used for public reporting of heart transplant center performance

Heart transplant programs and regulatory entities require highly accurate performance metrics to support internal quality improvement activities and national oversight of transplant programs, respectively. We assessed the accuracy of publicly reported performance measures. We used the United Network for Organ Sharing registry to study patients who underwent heart transplantation between January 1, 2016 and June 30, 2018. We used tests of calibration to compare the observed rate of 1-year graft failure to the expected risk of 1-year graft failure, which was calculated for each recipient using the July 2019 method published by the Scientific Registry of Transplant Recipients (SRTR). The primary study outcome was the joint test of calibration, which accounts for both the total number of events predicted (calibration-in-the-large) and dispersion of risk predictions (calibration slope). 6,528 heart transplants were analyzed. The primary test of calibration failed (p <0.0001), indicating poor accuracy of the SRTR model. The calibration-in-the-large statistic (0.63, 95% confidence interval [CI] 0.58-0.68, p < 0.0001) demonstrated overestimation of event rates while the calibration slope statistic (0.56, 95% CI 0.49-0.62, p <0.0001) indicated over-dispersion of event rates. Pre-specified subgroup analyses demonstrated poor calibration for all subgroups (each p <0.01). After recalibration, program-level observed/expected ratios increased by a median of 0.14 (p <0.0001). Risk models employed for publicly-reported graft survival at U.S. heart transplant centers lack accuracy in general and in all subgroups tested. The use of disease-specific models may improve the accuracy of program performance metrics.

Derivation and validation of predictors of oral anticoagulant-related adverse events in seniors transitioning from hospital to home

IntroductionOral anticoagulant (OAC)-related adverse events are high post-hospitalization. We planned to develop and validate a prediction model for OAC-related harm within 30 days of hospitalization. MethodsWe undertook a population-based study of adults aged ≥66 years who were discharged from hospital on an OAC from September 2010 to March 2015 in Ontario, Canada. The primary outcome was a composite of time to first hospitalization or emergency department visit for a hemorrhagic or thromboembolic event, or mortality within 30 days of hospital discharge. Cox proportional hazards regression was used to build the model. ResultsWe included 120,721 patients of which 5423 experienced the outcome. Most patients were aged ≥75 years (59.5%) and were female (55.6%). Sixty percent of the cohort had a follow-up visit with a healthcare provider within 7 days of discharge. Patients discharged on a direct acting OAC versus warfarin (apixaban: Hazard Ratio [HR] 0.82, 95% confidence interval [CI] 0.71-0.94; dabigatran: HR 0.73, 95% CI 0.63-0.84; rivaroxaban: HR 0.79, 95% CI 0.71-0.88), were prevalent users of the dispensed OAC versus incident users (HR 0.82, 95% CI 0.69-0.96), had a joint replacement in the past 35 days (HR 0.40, 95% CI 0.33-0.50) or major surgery during index hospital stay (HR 0.69, 95% CI 0.60-0.80) had a lower risk for the outcome. The Cox model was stable with acceptable discrimination but poor goodness-of-fit. ConclusionsA model for OAC-related harm in the early post-discharge period was developed. External validation studies are required to understand the model's poor calibration.

The Ontario Domestic Assault Risk Assessment: Predicting Violence Among Men With a Police Record of Intimate Partner Violence in the United States

The Ontario Domestic Assault Risk Assessment (ODARA) is an actuarial risk assessment tool for intimate partner violence (IPV) recidivism. Despite its international use, there is no published validation of the ODARA’s predictive accuracy in a U.S. sample. We studied 356 men in New York police records of IPV against a female partner to examine the ODARA’s predictive accuracy for IPV recidivism (base rate 35%), non-IPV violent recidivism (against a nonpartner; 16%), any violent recidivism (49%), and nonviolent recidivism (50%), in a fixed 2-year follow-up. Using 11 scorable ODARA items, area under the curve values were significant and ranged from .590 to .630, indicating small to medium effects. Expected/Observed indices revealed poor calibration with 2-year IPV recidivism rates in ODARA construction and cross-validation samples. Findings support the generalization of the ODARA’s predictive accuracy in different populations and outcomes, but a need for new norm development for higher risk populations.

Predicting outcomes of psychotherapy for depression with electronic health record data.

Predictive analytics with electronic health record (EHR) data holds promise for improving outcomes of psychiatric care. This study evaluated models for predicting outcomes of psychotherapy for depression in a clinical practice setting. EHR data from two large integrated health systems (Kaiser Permanente Colorado and Washington) included 5,554 new psychotherapy episodes with a baseline Patient Health Questionnaire (PHQ-9) score ≥ 10 and a follow-up PHQ-9 14–180 days after treatment initiation. Baseline predictors included demographics and diagnostic, medication, and encounter history. Prediction models for two outcomes—follow-up PHQ-9 score and treatment response (≥ 50% PHQ-9 reduction)—were trained in a random sample of 70% of episodes and validated in the remaining 30%. Two methods were used for modeling: generalized linear regression models with variable selection and random forests. Sensitivity analyses considered alternate predictor, outcome, and model specifications. Predictions of follow-up PHQ-9 scores poorly estimated observed outcomes (mean squared error = 31 for linear regression, 40 for random forest). Predictions of treatment response had low discrimination (AUC = 0.57 for logistic regression, 0.61 for random forest), low classification accuracy, and poor calibration. Sensitivity analyses showed similar results. We note that prediction model performance may vary for settings with different care or EHR documentation practices. In conclusion, prediction models did not accurately predict depression treatment outcomes despite using rich EHR data and advanced analytic techniques. Health systems should proceed cautiously when considering prediction models for psychiatric outcomes using baseline intake information. Transparent research should be conducted to evaluate performance of any model intended for clinical use.

Modelling hospital outcome: problems with endogeneity

BackgroundMortality modelling in the critical care paradigm traditionally uses logistic regression, despite the availability of estimators commonly used in alternate disciplines. Little attention has been paid to covariate endogeneity and the status of non-randomized treatment assignment. Using a large registry database, various binary outcome modelling strategies and methods to account for covariate endogeneity were explored.MethodsPatient mortality data was sourced from the Australian & New Zealand Intensive Society Adult Patient Database for 2016. Hospital mortality was modelled using logistic, probit and linear probability (LPM) models with intensive care (ICU) providers as fixed (FE) and random (RE) effects. Model comparison entailed indices of discrimination and calibration, information criteria (AIC and BIC) and binned residual analysis. Suspect covariate and ventilation treatment assignment endogeneity was identified by correlation between predictor variable and hospital mortality error terms, using the Stata™ “eprobit” estimator. Marginal effects were used to demonstrate effect estimate differences between probit and “eprobit” models.ResultsThe cohort comprised 92,693 patients from 124 intensive care units (ICU) in calendar year 2016. Patients mean age was 61.8 (SD 17.5) years, 41.6% were female and APACHE III severity of illness score 54.5(25.6); 43.7% were ventilated. Of the models considered in predicting hospital mortality, logistic regression (with or without ICU FE) and RE logistic regression dominated, more so the latter using information criteria indices. The LPM suffered from many predictions outside the unit [0,1] interval and both poor discrimination and calibration. Error terms of hospital length of stay, an independent risk of death score and ventilation status were correlated with the mortality error term. Marked differences in the ventilation mortality marginal effect was demonstrated between the probit and the "eprobit" models which were scenario dependent. Endogeneity was not demonstrated for the APACHE III score.ConclusionsLogistic regression accounting for provider effects was the preferred estimator for hospital mortality modelling. Endogeneity of covariates and treatment variables may be identified using appropriate modelling, but failure to do so yields problematic effect estimates.

External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients

The Epic Sepsis Model (ESM), a proprietary sepsis prediction model, is implemented at hundreds of US hospitals. The ESM's ability to identify patients with sepsis has not been adequately evaluated despite widespread use. To externally validate the ESM in the prediction of sepsis and evaluate its potential clinical value compared with usual care. This retrospective cohort study was conducted among 27 697 patients aged 18 years or older admitted to Michigan Medicine, the academic health system of the University of Michigan, Ann Arbor, with 38 455 hospitalizations between December 6, 2018, and October 20, 2019. The ESM score, calculated every 15 minutes. Sepsis, as defined by a composite of (1) the Centers for Disease Control and Prevention surveillance criteria and (2) International Statistical Classification of Diseases and Related Health Problems, Tenth Revision diagnostic codes accompanied by 2 systemic inflammatory response syndrome criteria and 1 organ dysfunction criterion within 6 hours of one another. Model discrimination was assessed using the area under the receiver operating characteristic curve at the hospitalization level and with prediction horizons of 4, 8, 12, and 24 hours. Model calibration was evaluated with calibration plots. The potential clinical benefit associated with the ESM was assessed by evaluating the added benefit of the ESM score compared with contemporary clinical practice (based on timely administration of antibiotics). Alert fatigue was evaluated by comparing the clinical value of different alerting strategies. We identified 27 697 patients who had 38 455 hospitalizations (21 904 women [57%]; median age, 56 years [interquartile range, 35-69 years]) meeting inclusion criteria, of whom sepsis occurred in 2552 (7%). The ESM had a hospitalization-level area under the receiver operating characteristic curve of 0.63 (95% CI, 0.62-0.64). The ESM identified 183 of 2552 patients with sepsis (7%) who did not receive timely administration of antibiotics, highlighting the low sensitivity of the ESM in comparison with contemporary clinical practice. The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue. This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis. The widespread adoption of the ESM despite its poor performance raises fundamental concerns about sepsis management on a national level.

Avoiding Unnecessary Biopsy: MRI-based Risk Models versus a PI-RADS and PSA Density Strategy for Clinically Significant Prostate Cancer.

Background In validation studies, risk models for clinically significant prostate cancer (csPCa; Gleason score ≥3+4) combining multiparametric MRI and clinical factors have demonstrated poor calibration (over- and underprediction) and limited use in avoiding unnecessary prostate biopsies. Purpose MRI-based risk models following local recalibration were compared with a strategy that combined Prostate Imaging Data and Reporting System (PI-RADS; version 2) and prostate-specific antigen density (PSAd) to assess the potential reduction of unnecessary prostate biopsies. Materials and Methods This retrospective study included 385 patients without prostate cancer diagnosis who underwent multipara-metric MRI (PI-RADS category ≥3) and MRI-targeted biopsy between 2015 and 2019. Recalibration and selection of the best-performing MRI model (MRI-European Randomized Study of Screening for Prostate Cancer [ERSPC], van Leeuwen, Radtke, and Mehralivand models) were undertaken in cohort C1 (n = 242; 2015-2017). The impact on biopsy decisions was compared with an alternative strategy (no biopsy for PI-RADS category 3 plus PSAd < 0.1 ng/mL per milliliter) in cohort C2 (n = 143; 2018-2019). Discrimination, calibration, and clinical utility were assessed by using the area under the receiver operating characteristic curve (AUC), calibration plots, and decision curve analysis, respectively. Results The prevalence of csPCa was 38% (93 of 242 patients) and 45% (64 of 143 patients) in cohorts C1 and C2, respectively. Decision curve analysis demonstrated the highest net benefit for the van Leeuwen and Mehralivand models in C1. Used for biopsy decisions in C2, van Leeuwen (AUC, 0.84; 95% CI: 0.77, 0.9) and Mehralivand (AUC, 0.79; 95% CI: 0.72, 0.86) enabled no net benefit at a risk threshold of 10%. Up to a risk threshold of 15%, net benefit remained inferior to the PI-RADS plus PSAd strategy, which avoided biopsy in 63 per 1000 men, without missing csPCa. Without prior recalibration in C1, three of four models (MRIERSPC, Radtke, Mehralivand) were poorly calibrated and not clinically useful in C2. Conclusion The number of unnecessary prostate biopsies in men with positive MRI may be safely reduced by using a prostate-specific antigen density-based strategy. In a risk-averse scenario, this strategy enabled better biopsy decisions compared with MRI-based risk models. ©RSNA, 2021 Online supplemental material is available for this article.

Rotterdam mobile phone app including MRI data for the prediction of prostate cancer: A multicenter external validation

ObjectivesThe Rotterdam Prostate Cancer Risk calculator (RPCRC) has been validated in the past years. Recently a new version including multiparametric magnetic resonance imaging (mpMRI) data has been released. The aim of our study was to analyze the performance of the mpMRI RPCRC app. MethodsA series of men undergoing prostate biopsies were enrolled in eleven Italian centers. Indications for prostate biopsy included: abnormal Prostate specific antigen levels (PSA>4 ng/ml), abnormal DRE and abnormal mpMRI. Patients’ characteristics were recorded. Prostate cancer (PCa) risk and high-grade PCa risk were assessed using the RPCRC app. The performance of the mpMRI RPCRC in the prediction of cancer and high-grade PCa was evaluated using receiver operator characteristics, calibration plots and decision curve analysis. ResultsOverall, 580 patients were enrolled: 404/580 (70%) presented PCa and out of them 224/404 (55%) presented high-grade PCa. In the prediction of cancer, the RC presented good discrimination (AUC = 0.74), poor calibration (p = 0.01) and a clinical net benefit in the range of probabilities between 50 and 90% for the prediction of PCa (Fig. 1). In the prediction of high-grade PCa, the RC presented good discrimination (AUC = 0.79), good calibration (p = 0.48) and a clinical net benefit in the range of probabilities between 20 and 80% (Fig. 1). ConclusionsThe Rotterdam prostate cancer risk App accurately predicts the risk of PCa and particularly high-grade cancer. The clinical net benefit is wide for high-grade cancer and therefore its implementation in clinical practice should be encouraged. Further studies should assess its definitive role in clinical practice.