Re: External validation of HAS model in predicting mortality after emergency laparotomy.

Aims The HAS (Hajibandeh Index, American Society of Anaesthesiologists status, and sarcopenia) model has demonstrated superior predictive ability compared to the NELA (National Emergency Laparotomy Audit) score for mortality after emergency laparotomy. However, external validation of this model is lacking. This study aimed to validate the HAS model's performance in predicting mortality after emergency laparotomy and compare its predictive accuracy with the NELA score. Methods This retrospective cohort study included all adult patients (>17 years) who underwent emergency laparotomy between January 2022 and June 2023 at an urban district general hospital. Receiver operating characteristic curve analysis was used to compare the HAS model and NELA score in predicting 30-day mortality. Subgroup analysis was conducted for age groups: ≥50, ≥60, ≥70, and ≥80. Results A total of 117 patients were included. The area under the curve (AUC) for the HAS model in predicting 30-day mortality was 0.90 (95% CI 0.83-0.95). There was no statistically significant difference in AUC between the HAS and NELA scores for all patients (0.90 vs 0.80, p = 0.097). However, the AUC of the HAS model was significantly superior to that of the NELA score in patients aged ≥50 (0.89 vs 0.75, p = 0.040), ≥60 (0.87 vs 0.69, p = 0.020), ≥70 (0.85 vs 0.67, p = 0.030), and ≥80 (0.90 vs 0.66, p < 0.001). Conclusions This study provides external validation of the HAS model for predicting 30-day mortality after emergency laparotomy. Prospective studies with larger sample sizes are warranted to confirm these findings.

Aim. To perform external validation of a multivariate model for predicting the risk of death in patients with an implantable cardioverter – defibrillator (ICD) in an independent sample. Materials and methods. The group for model development included 260 patients from the Implantable Cardioverter – Defibrillator Patient Registry who had an ICD implanted between 2015 and 2019. External validation of the model was carried out in an independent, prospective, observational cohort study of patients from the same registry, in whom an ICD was implanted between 2020 and 2021, a total of 94 patients, median age 66 (52;73) years, 73 (77.6%) men, 21 (22.4%) women. In 89 (94.7%) patients, an ICD was implanted for primary prevention of sudden cardiac death. Following a telephone survey and examination of medical records from hospital and clinic databases, data on the vital status (alive / dead) and causes of death were obtained during a 2.5-year follow-up. The actual and predicted mortality from the estimated multivariate model were compared. Results. During the follow-up, a total of 26 (27.7%) patients died in the external validation group, which was comparable to the development group (p > 0.05). In the group of deceased, 15 (57.7%) people developed acute decompensated heart failure, 4 (14.8%) had myocardial infarction, 6 (23.1%) had pneumonia caused by a new coronavirus infection, and one (3.8%) patient died due to an infectious complication. The diagnostic accuracy of the multivariate model for predicting the risk of death in patients with ICD in an independent sample was sufficient (the area under the curve (AUC) of the created model was 0.8). The sensitivity of the model was 76.2%, specificity – 76.1%. Previously, in the development cohort, AUC of the created model was 0.8, the sensitivity of the model was 75.7%, and the specificity was 80%. Model significance did not differ significantly between the development and external validation groups (p = 0.102, McNeil test).Conclusion. The multivariate prediction model has sufficient statistical power to predict the risk of long-term death after ICD implantation, which was externally validated.

Precise and timely referral for lung transplantation is critical for the survival of cystic fibrosis patients with terminal illness. While machine learning (ML) models have been shown to achieve significant improvement in prognostic accuracy over current referral guidelines, the external validity of these models and their resulting referral policies has not been fully investigated. Here, we studied the external validity of machine learning-based prognostic models using annual follow-up data from the UK and Canadian Cystic Fibrosis Registries. Using a state-of-the-art automated ML framework, we derived a model for predicting poor clinical outcomes in patients enrolled in the UK registry, and conducted external validation of the derived model using the Canadian Cystic Fibrosis Registry. In particular, we studied the effect of (1) natural variations in patient characteristics across populations and (2) differences in clinical practice on the external validity of ML-based prognostic scores. Overall, decrease in prognostic accuracy on the external validation set (AUCROC: 0.88, 95% CI 0.88-0.88) was observed compared to the internal validation accuracy (AUCROC: 0.91, 95% CI 0.90-0.92). Based on our ML model, analysis on feature contributions and risk strata revealed that, while external validation of ML models exhibited high precision on average, both factors (1) and (2) can undermine the external validity of ML models in patient subgroups with moderate risk for poor outcomes. A significant boost in prognostic power (F1 score) from 0.33 (95% CI 0.31-0.35) to 0.45 (95% CI 0.45-0.45) was observed in external validation when variations in these subgroups were accounted in our model. Our study highlighted the significance of external validation of ML models for cystic fibrosis prognostication. The uncovered insights on key risk factors and patient subgroups can be used to guide the cross-population adaptation of ML-based models and inspire new research on applying transfer learning methods for fine-tuning ML models to cope with regional variations in clinical care.

Prediction models for COVID-19 clinical decision making

Temporal and external validation of the fullPIERS model for the prediction of adverse maternal outcomes in women with pre-eclampsia.

Aim. Development and external validation of a risk prediction model for acute decompensated heart failure (ADHF) in patients with low left ventricular ejection fraction.Material and methods. The model development group was represented by patients with heart failure with reduced ejection fraction (HFrEF) included in a registry observational study from 2015 to 2019, a total of 260 patients, age 59 (53; 66) years, 214 (82.3%) — men. External validation of the model was carried out in a cohort of independent prospective observation of 94 patients with HFrEF from the same registry for the period from 2020 to 2021, median age 66 (52;73) years, of which 73 (77.6%) were men. The prospective follow-up period was 4.6 (2.3; 4.9) years in the internal validation group, 2.5 (1.7; 2.9) years in the external validation group. Data were obtained on the status of patients, causes of death, and the frequency of hospitalizations for ADHF. The actual and predicted incidence of ADHF using the evaluated prognostic model was compared.Results. During the observation period in the internal validation group, ADHF developed in 69 (26.5%) patients, and 47 (18.1%) died due to ADHF. The prognostic regression model included LA enlargement of more than 45 mm, male gender, left ventricular ejection fraction less than 35%, absence of renin-angiotensin system blocker and amiodarone. When performing ROC analysis, the area under the ROC curve (AUC) of the created model was 0.8, sensitivity model — 69.2%, specificity — 80%, accuracy — 75.3%. In the external validation group, 34 (36.2%) cases of ADHF were registered; mortality from ADHF in the external validation group was 15.9%, which is comparable to the development group (p > 0.05). The diagnostic value of the developed model during external validation showed to be high and was comparable to the results obtained in the development group: the area under the ROC curve (AUC) was 0.8, sensitivity — 73.3%, specificity — 82.5%, accuracy 76.1%, (p=0.102, McNeil test).Conclusion. The developed regression model has sufficient statistical power to predict the risk of ADHF in patients with low left ventricular ejection fraction in the long term, which is confirmed by external validation.

Response prediction for neoadjuvant treatment in locally advanced rectal cancer patients-improvement in decision-making: a systematic review

IntroductionWe aimed to externally validate the performance of the HAS model (Hajibandeh Index, American Society of Anaesthesiologists status, and sarcopenia) in predicting mortality after emergency laparotomy. We also aimed to compare the HAS model with the Parsimonious NELA (National Emergency Laparotomy Audit) risk score.MethodsIn this retrospective cohort study, we included adult patients who underwent emergency laparotomy between January 2022 and June 2023. The performance of the HAS score and the NELA score in predicting 30-day mortality was compared using receiver operating characteristic (ROC) curve analysis. We performed subgroup analysis for the following age groups: age ≥50, age ≥60, age ≥70, and age ≥80 years.FindingsWe included 117 patients in this study. ROC curve analysis showed that area under the curve (AUC) of the HAS score for 30-day mortality was 0.90 (95% CI 0.83–0.95). Although the AUC of HAS score was higher than the AUC of NELA score for all patients, this was not statistically significant (0.90 vs 0.80, p=0.097). AUC of the HAS score was superior to NELA score in patients aged ≥50 (0.89 vs 0.75, p=0.040), patients aged ≥60 (0.87 vs 0.69, p=0.020), patients aged ≥70 (0.85 vs 0.67, p=0.030), and patients aged ≥80 (0.90 vs 0.66, p<0.001).ConclusionsThe results of the current study support the external validity of the HAS model in predicting 30-day mortality after emergency laparotomy. Prospective studies with larger sample size are required.

Simple SummaryUp to 30% of patients develop severe complications following pancreatoduodenectomy (PD). With respect to risk stratification and shared decision making, prediction models to predict complications are crucial. In 2015, a risk model for severe complications was developed by Schroder et al. based on three preoperative variables: BMI, ASA classification and mean Hounsfield Units of the pancreatic body on the preoperative abdominal CT scan. However, external validation of this model has not yet been performed. It is important to validate prediction models externally before implementing them in clinical practice to confirm their accuracy and generalizability when applied to a different patient population. Our aim was to externally validate this risk prediction model using an independent cohort of patients.Background: Pancreatoduodenectomy (PD) is the only cure for periampullary and pancreatic cancer. It has morbidity rates of 40–60%, with severe complications in 30%. Prediction models to predict complications are crucial. A risk model for severe complications was developed by Schroder et al. based on BMI, ASA classification and Hounsfield Units of the pancreatic body on the preoperative CT scan. These variables were independent predictors for severe complications upon internal validation. Our aim was to externally validate this model using an independent cohort of patients. Methods: A retrospective analysis was performed on 318 patients who underwent PD at our institution from 2013 to 2021. The outcome of interest was severe complications Clavien–Dindo ≥ IIIa. Model calibration, discrimination and performance were assessed. Results: A total of 308 patients were included. Patients with incomplete data were excluded. A total of 89 (28.9%) patients had severe complications. The externally validated model achieved: C-index = 0.67 (95% CI: 0.60–0.73), regression coefficient = 0.37, intercept = 0.13, Brier score = 0.25. Conclusions: The performance ability, discriminative power, and calibration of this model were acceptable. Our risk calculator can help surgeons identify high-risk patients for post-operative complications to improve shared decision-making and tailor perioperative management.

In a recent scoping review, we identified 43 mortality prediction models for critically ill patients. We aimed to assess the performances of these models through external validation. Multicenter study. External validation of models was performed in the Simple Intensive Care Studies-I (SICS-I) and the Finnish Acute Kidney Injury (FINNAKI) study. The SICS-I study consisted of 1,075 patients, and the FINNAKI study consisted of 2,901 critically ill patients. For each model, we assessed: 1) the original publications for the data needed for model reconstruction, 2) availability of the variables, 3) model performance in two independent cohorts, and 4) the effects of recalibration on model performance. The models were recalibrated using data of the SICS-I and subsequently validated using data of the FINNAKI study. We evaluated overall model performance using various indexes, including the (scaled) Brier score, discrimination (area under the curve of the receiver operating characteristics), calibration (intercepts and slopes), and decision curves. Eleven models (26%) could be externally validated. The Acute Physiology And Chronic Health Evaluation (APACHE) II, APACHE IV, Simplified Acute Physiology Score (SAPS)-Reduced (SAPS-R)' and Simplified Mortality Score for the ICU models showed the best scaled Brier scores of 0.11' 0.10' 0.10' and 0.06' respectively. SAPS II, APACHE II, and APACHE IV discriminated best; overall discrimination of models ranged from area under the curve of the receiver operating characteristics of 0.63 (0.61-0.66) to 0.83 (0.81-0.85). We observed poor calibration in most models, which improved to at least moderate after recalibration of intercepts and slopes. The decision curve showed a positive net benefit in the 0-60% threshold probability range for APACHE IV and SAPS-R. In only 11 out of 43 available mortality prediction models, the performance could be studied using two cohorts of critically ill patients. External validation showed that the discriminative ability of APACHE II, APACHE IV, and SAPS II was acceptable to excellent, whereas calibration was poor.

External validation of a lung cancer-based prediction model for two-year mortality in esophageal cancer patient cohorts

ObjectiveTo develop and externally validate a prognostic model built on important factors predisposing multimorbid patients to all-cause readmission and/or death. In addition to identify patients who may benefit most from a comprehensive clinical pharmacist intervention. MethodsA multivariable prognostic model was developed based on data from a randomised controlled trial investigating the effect of pharmacist-led medicines management on readmission rate in multimorbid, hospitalised patients. The derivation set comprised 386 patients randomised in a 1:1 manner to the intervention group, i.e. with a pharmacist included in their multidisciplinary treatment team, or the control group receiving standard care at the ward. External validation of the model was performed using data from an independent cohort, in which 100 patients were randomised to the same intervention, or standard care. The setting was an internal medicines ward at a university hospital in Norway. ResultsThe number of patients who were readmitted or had died within 18 months after discharge was 297 (76.9 %) in the derivation set, i.e. the randomized controlled trial, and 69 (71.1 %) in the validation set, i.e. the independent cohort. Charlson comorbidity index (CCI; low, moderate or high), previous hospital admissions within the previous six months and heart failure were the strongest prognostic factors and were included in the final model. The efficacy of the pharmaceutical intervention did not prove significant in the model. A prognostic index (PI) was constructed to estimate the hazard of readmission or death (low, intermediate or high-risk groups). Overall, the external validation replicated the result. We were unable to identify a subgroup of the multimorbid patients with better efficacy of the intervention. ConclusionsA prognostic model including CCI, previous admissions and heart failure can be used to obtain valid estimates of risk of readmission and death in patients with multimorbidity.

IntroductionThe Early Prediction of Functional Outcome after Stroke (EPOS) model for independent gait is a tool to predict between days 2 and 9 poststroke whether patients will regain independent gait 6 months after stroke. External validation of the model is important to determine its clinical applicability and generalizability by testing its performance in an independent cohort. Therefore, this study aimed to perform a temporal and geographical external validation of the EPOS prediction model for independent gait after stroke but with the endpoint being 3 months instead of the original 6 months poststroke.MethodsTwo prospective longitudinal cohort studies consisting of patients with first-ever stroke admitted to a Swiss hospital stroke unit. Sitting balance and strength of the paretic leg were tested at days 1 and 8 post-stroke in Cohort I and at days 3 and 9 in Cohort II. Independent gait was assessed 3 months after symptom onset. The performance of the model in terms of discrimination (area under the receiver operator characteristic (ROC) curve; AUC), classification, and calibration was assessed.ResultsIn Cohort I [N = 39, median age: 74 years, 33% women, median National Institutes of Health Stroke Scale (NIHSS) 9], the AUC (95% confidence interval (CI)] was 0.675 (0.510, 0.841) on day 1 and 0.921 (0.811, 1.000) on day 8. For Cohort II (N = 78, median age: 69 years, 37% women, median NIHSS 8), this was 0.801 (0.684, 0.918) on day 3 and 0.846 (0.741, 0.951) on day 9.Discussion and ConclusionExternal validation of the EPOS prediction model for independent gait 3 months after stroke resulted in an acceptable performance from day 3 onward in mild-to-moderately affected patients with first-ever stroke without severe prestroke disability. The impact of applying this model in clinical practice should be investigated within this subgroup of patients with stroke. To improve the generalizability of patients with recurrent stroke and those with more severe, neurological comorbidities, the performance of the EPOS model within these patients should be determined across different geographical areas.

Development and external validation of a population pharmacokinetic model and optimal Vancomycin dosing regimen for overweight and obese patients.

BackgroundPneumonia remains difficult to diagnose in primary care. Prediction models based on signs and symptoms (S&S) serve to minimize the diagnostic uncertainty. External validation of these models is essential before implementation into routine practice. In this study all published S&S models for prediction of pneumonia in primary care were externally validated in the individual patient data (IPD) of previously performed diagnostic studies.Methods and FindingsS&S models for diagnosing pneumonia in adults presenting to primary care with lower respiratory tract infection and IPD for validation were identified through a systematical search. Six prediction models and IPD of eight diagnostic studies (N total = 5308, prevalence pneumonia 12%) were included. Models were assessed on discrimination and calibration. Discrimination was measured using the pooled Area Under the Curve (AUC) and delta AUC, representing the performance of an individual model relative to the average dataset performance. Prediction models by van Vugt et al. and Heckerling et al. demonstrated the highest pooled AUC of 0.79 (95% CI 0.74–0.85) and 0.72 (0.68–0.76), respectively. Other models by Diehr et al., Singal et al., Melbye et al., and Hopstaken et al. demonstrated pooled AUCs of 0.65 (0.61–0.68), 0.64 (0.61–0.67), 0.56 (0.49–0.63) and 0.53 (0.5–0.56), respectively. A similar ranking was present based on the delta AUCs of the models. Calibration demonstrated close agreement of observed and predicted probabilities in the models by van Vugt et al. and Singal et al., other models lacked such correspondence. The absence of predictors in the IPD on dataset level hampered a systematical comparison of model performance and could be a limitation to the study.ConclusionsThe model by van Vugt et al. demonstrated the highest discriminative accuracy coupled with reasonable to good calibration across the IPD of different study populations. This model is therefore the main candidate for primary care use.