External Validation Of Prediction Model Research Articles

Independent external validation of predictive models for urinary dysfunction following external beam radiotherapy of the prostate: Issues in model development and reporting

Background and purposeMost predictive models are not sufficiently validated for prospective use. We performed independent external validation of published predictive models for urinary dysfunctions following radiotherapy of the prostate. Materials/methodsMultivariable models developed to predict atomised and generalised urinary symptoms, both acute and late, were considered for validation using a dataset representing 754 participants from the TROG 03.04-RADAR trial. Endpoints and features were harmonised to match the predictive models. The overall performance, calibration and discrimination were assessed. Results14 models from four publications were validated. The discrimination of the predictive models in an independent external validation cohort, measured using the area under the receiver operating characteristic (ROC) curve, ranged from 0.473 to 0.695, generally lower than in internal validation. 4 models had ROC >0.6. Shrinkage was required for all predictive models’ coefficients ranging from −0.309 (prediction probability was inverse to observed proportion) to 0.823. Predictive models which include baseline symptoms as a feature produced the highest discrimination. Two models produced a predicted probability of 0 and 1 for all patients. ConclusionsPredictive models vary in performance and transferability illustrating the need for improvements in model development and reporting. Several models showed reasonable potential but efforts should be increased to improve performance. Baseline symptoms should always be considered as potential features for predictive models.

Acute kidney injury following liver transplantation: a systematic review of published predictive models.

Acute kidney injury is a frequent postoperative complication amongst liver transplant recipients and is associated with increased morbidity and mortality. This systematic review analysed the existing predictive models, in order to solidify current understanding. Articles were selected for inclusion if they described the primary development of a clinical prediction model (either an algorithm or risk score) to predict AKI post liver transplantation. The database search yielded a total of seven studies describing the primary development of a prediction model or risk score for the development of AKI following liver transplantation. The models span thirteen years of clinical research and highlight a gradual change in the definitions of AKI, emphasising the need to employ standardised definitions for subsequent studies. Collectively, the models identify a diverse range of predictive factors with several common trends. They emphasise the impact of preoperative renal dysfunction, liver disease severity and aetiology, metabolic risk factors as well as intraoperative variables including measures of haemodynamic instability and graft quality. Although several of the models address postoperative parameters, their utility in predictive modelling seems to be of questionable relevance. The common risk factors identified within this systematic review provide a minimum list of variables, which future studies should address. Research in this area would benefit from prospective, multi-site studies with larger cohorts as well as the subsequent internal and external validation of predictive models. Ultimately, the ability to identify patients at high risk of post-transplant AKI may enable early intervention and perhaps prevention.

External Validation of Prediction Models for Pneumonia in Primary Care Patients with Lower Respiratory Tract Infection: An Individual Patient Data Meta-Analysis.

BackgroundPneumonia remains difficult to diagnose in primary care. Prediction models based on signs and symptoms (S&S) serve to minimize the diagnostic uncertainty. External validation of these models is essential before implementation into routine practice. In this study all published S&S models for prediction of pneumonia in primary care were externally validated in the individual patient data (IPD) of previously performed diagnostic studies.Methods and FindingsS&S models for diagnosing pneumonia in adults presenting to primary care with lower respiratory tract infection and IPD for validation were identified through a systematical search. Six prediction models and IPD of eight diagnostic studies (N total = 5308, prevalence pneumonia 12%) were included. Models were assessed on discrimination and calibration. Discrimination was measured using the pooled Area Under the Curve (AUC) and delta AUC, representing the performance of an individual model relative to the average dataset performance. Prediction models by van Vugt et al. and Heckerling et al. demonstrated the highest pooled AUC of 0.79 (95% CI 0.74–0.85) and 0.72 (0.68–0.76), respectively. Other models by Diehr et al., Singal et al., Melbye et al., and Hopstaken et al. demonstrated pooled AUCs of 0.65 (0.61–0.68), 0.64 (0.61–0.67), 0.56 (0.49–0.63) and 0.53 (0.5–0.56), respectively. A similar ranking was present based on the delta AUCs of the models. Calibration demonstrated close agreement of observed and predicted probabilities in the models by van Vugt et al. and Singal et al., other models lacked such correspondence. The absence of predictors in the IPD on dataset level hampered a systematical comparison of model performance and could be a limitation to the study.ConclusionsThe model by van Vugt et al. demonstrated the highest discriminative accuracy coupled with reasonable to good calibration across the IPD of different study populations. This model is therefore the main candidate for primary care use.

441: A systematic review and external validation of prediction models for pregnancy complications: the RESPECT study

441: A systematic review and external validation of prediction models for pregnancy complications: the RESPECT study

First trimester screening of late-onset preeclampsia in a low risk and low volume obstetrical setting: external validation of a predictive model

The external validation of predictive model of late preeclampsia in a low volume and low risk obstetrical setting. A cohort was created of 174 singleton pregnancies of 11+0-13+6 gestational weeks at Clinica Universidad Navarra from September 2011 to March 2013, which was considered as a validation cohort of a previously described model for late PE (Hospital Clinic, Barcelona). A total of 7 (4%) women developed late PE. In the validation cohort the area under the curve of the model was 0.69 (95% CI 0.45-0.93). Detection rates for a 5, 10 and 15% of false positive rates were 21.9, 31.4% and 38.6% respec-tively. When comparing the areas under the curve of the validation cohort with the construction cohort, no statistical differences were found (p=0.68). The combination of maternal history, pregnancy associated plasma protein-A and mean arterial pressure is moderately useful to predict preeclampsia in a low risk and low volume obstetrical setting. The predictive model of the Clinic Hospital of Barcelona is a valid tool in predicting late preeclampsia in this setting.

Screening de primer trimestre de preeclampsia tardía en gestantes de bajo riesgo en un centro de bajo volumen obstétrico: validación externa de un modelo predictivo

Background. The external validation of predictive model of late preeclampsia in a low volume and low risk obstetrical setting. Methods. A cohort was created of 174 singleton pregnancies of 11 +0 -13 +6 gestational weeks at Clinica Universidad Navarra from September 2011 to March 2013, which was considered as a validation cohort of a previously described model for late PE (Hospital Clinic, Barcelona). Results. A total of 7 (4%) women developed late PE. In the validation cohort the area under the curve of the model was 0.69 (95% CI 0.45-0.93). Detection rates for a 5, 10 and 15% of false positive rates were 21.9, 31.4% and 38.6% respectively. When comparing the areas under the curve of the validation cohort with the construction cohort, no statistical differences were found (p=0.68). Conclusion. The combination of maternal history, pregnancy associated plasma protein-A and mean arterial pressure is moderately useful to predict preeclampsia in a low risk and low volume obstetrical setting. The predictive model of the Clinic Hospital of Barcelona is a valid tool in predicting late preeclampsia in this setting.

Antimüllerian hormone levels and antral follicle count as prognostic indicators in a personalized prediction model of live birth

To compare antimüllerian hormone (AMH) and antral follicle count (AFC) separately and in combination with clinical characteristics for the prediction of live birth after controlled ovarian stimulation. Retrospective development and temporal external validation of prediction model. Outpatient IVF clinic. We applied the boosted tree method to develop three prediction models incorporating clinical characteristics plus AMH or AFC or the combination on 2,124 linked IVF cycles from 2006 to 2010 and temporally externally validated predicted live-birth probabilities with an independent data set comprising 1,121 cycles from 2011 to 2012. None. Predictive power (posterior log of odds ratio compared to age, or PLORA), reclassification, receiver operator characteristic analysis, calibration, dynamic range. Predictive power, was highest for the AMH model (PLORA = 29.1), followed by the AMH-AFC model (PLORA = 28.3) and AFC model (PLORA = 22.5). The prediction errors were 1% to <5% in each prognostic tier for all three models, except for the predicted live-birth probabilities of <10% in the AFC model, where the prediction error was 8%. The improvement in predictive power was highest for the AMH model: 76.2% improvement over age alone relative to 59% improvement for AFC and 73.3% for the combined model. Receiver operating characteristic analysis demonstrated that the AMH and the combined model had comparable discrimination (area under the curve = 0.716) and similar prediction error for high and low strata of live-birth prediction, with an improvement of 6.3% over age alone. The validated prediction model confirmed that AMH when combined with clinical characteristics can accurately identify the likelihood of live birth with a low prediction error. AFC provided no added predictive value beyond AMH.

A new framework to enhance the interpretation of external validation studies of clinical prediction models

ObjectivesIt is widely acknowledged that the performance of diagnostic and prognostic prediction models should be assessed in external validation studies with independent data from “different but related” samples as compared with that of the development sample. We developed a framework of methodological steps and statistical methods for analyzing and enhancing the interpretation of results from external validation studies of prediction models. Study Design and SettingWe propose to quantify the degree of relatedness between development and validation samples on a scale ranging from reproducibility to transportability by evaluating their corresponding case-mix differences. We subsequently assess the models' performance in the validation sample and interpret the performance in view of the case-mix differences. Finally, we may adjust the model to the validation setting. ResultsWe illustrate this three-step framework with a prediction model for diagnosing deep venous thrombosis using three validation samples with varying case mix. While one external validation sample merely assessed the model's reproducibility, two other samples rather assessed model transportability. The performance in all validation samples was adequate, and the model did not require extensive updating to correct for miscalibration or poor fit to the validation settings. ConclusionThe proposed framework enhances the interpretation of findings at external validation of prediction models.

Reply

We agree with Schoorel et al. that the area under the receiver–operating characteristics curve (AUC) for the model described in our paper is particularly high after inclusion of scar information. We were careful to state this in our paper: ‘The AUC for our model is perhaps surprisingly high. This may reflect the relatively small number of women in the study.’1. However, we also say in the paper that, even if we interpret this with caution, it suggests that the scar is a major factor in the behavior of the uterus during a trial of labor. We are unable to agree that it is not plausible that the morphology of the scar may be an important predictor of uterine behavior in labor. The main indications for repeat Cesarean section in the failed vaginal birth after Cesarean (VBAC) group in our paper were fetal distress and failure to progress. It is entirely possible that both these factors are exacerbated by poor scar healing, which in turn may be reflected in the morphological features of a scar on ultrasound. Furthermore, a poorly healed scar may impact on decidualization and subsequent placentation and be associated with a greater likelihood of fetal distress in labor. We investigated prospectively the changes in Cesarean section scar dimensions during pregnancy as well as the influence of previous VBAC on those changes2. Schoorel et al. are incorrect in their assertion that this paper contains an analysis relating scar features to the probability of a successful trial of VBAC. It does not. We do comment on residual myometrial thickness (RMT) in the small number of cases of scar rupture or dehiscence. This is because, after peer review of our paper, we were asked by the Editor of the Journal to insert some brief information on the RMT and delta RMT in cases of uterine rupture or dehiscence to give more clinical relevance to the paper, to which we agreed. However, an association between RMT and successful VBAC in the current study was not investigated prior to variable selection. It is also important to note that third-trimester sonographic measurements of the thickness of the lower uterine segment and scar features have been reported as a predictor of vaginal delivery3. Accordingly, this variable would be considered as a possible factor in any predictive model on the basis of expert knowledge and was the basis of our hypothesis when we planned the study. Hence, we used a priori variable selection based on knowledge, which is in agreement with contemporary methodological guidelines. Having published our data on interobserver variability, we are aware that measurements of RMT may impact on the performance of any model that uses this variable; to what extent this occurs will only become apparent when RMT is measured by other examiners4. We are pleased that Schoorel et al. agree with the emphasis we placed in our paper on the importance of external validation of prediction models, which is a vital step before considering the introduction of such models into clinical practice. As we state clearly in the paper: ‘It is important to highlight that whilst this model and approach show promise, external validation in different units and populations is needed before introduction into clinical practice.’ O. Naji†‡, L. Wynants§¶, A. Smith†, Y. Abdallah†‡, C. Stalder†, A. Sayasneh†‡, A. McIndoe†, S. Ghaem-Maghami*‡, S. Van Huffel§¶, B. Van Calster**, D. Timmerman** and T. Bourne†‡** †Obstetrics and Gynaecology Unit, Queen Charlotte's and Chelsea Hospital, Imperial College, London,UK; ‡Institute of Reproductive and Developmental Biology, Imperial College, London, UK; §Department of Electrical Engineering (ESAT-SCD), KU Leuven, Leuven, Belgium; ¶iMinds Future Health Department, KU Leuven–University of Leuven, Leuven, Belgium; **Department of Development and Regeneration, University Hospitals, KU Leuven, Leuven, Belgium *Correspondence. (e-mail: s.ghaem-maghami@imperial.ac.uk)

The Kaiser Permanente inpatient risk adjustment methodology was valid in an external patient population

Objectives Accurately predicting hospital mortality is necessary to measure and compare patient care. External validation of predictive models is required to truly prove their utility. This study assessed the Kaiser Permanente inpatient risk adjustment methodology for hospital mortality in a patient population distinct from that used for its derivation. Study Design and Setting Retrospective cohort study at two hospitals in Ottawa, Canada, involving all inpatients admitted between January 1998 and April 2002 ( n = 188,724). Statistical models for inpatient mortality were derived on a random half of the cohort and validated on the other half. Results Inpatient mortality was 3.3%. The model using original parameter estimates had excellent discrimination (c-statistic 89.4, 95% confidence interval [CI] 0.891–0.898) but poor calibration. Using data-based parameter estimates, discrimination was excellent (c-statistic 0.915, 95% CI 0.912–0.918) and remained so when patient comorbidity was expressed in the model using the Elixhauser Index (0.901, 0.898–0.904) or the Charlson Index (0.894, 0.891–0.897). These models accurately predicted the risk of hospital death. Conclusion The Kaiser Permanente inpatient risk adjustment methodology is a valid model for predicting hospital mortality risk. It performed equally well regardless of methods used to summarize patient comorbidity.

Internal and external validation of predictive models: A simulation study of bias and precision in small samples

We performed a simulation study to investigate the accuracy of bootstrap estimates of optimism (internal validation) and the precision of performance estimates in independent validation samples (external validation). We combined two data sets containing children presenting with fever without source ( n = 376 + 179 = 555; 120 bacterial infections). Random samples were drawn from this combined data set for the development ( n = 376) and validation ( n = 179) of logistic regression models. The models included statistically significant predictors for infection selected from a set of 57 candidate predictors. Model development, including the selection of predictors, and validation were repeated in a bootstrapping procedure. The resulting expected optimism estimate in the receiver operating characteristic (ROC) area was compared with the observed optimism according to independent validation samples. The average apparent ROC area was 0.74, which was expected (based on bootstrapping) to decrease by 0.07 to 0.67, whereas the observed decrease in the validation samples was 0.09 to 0.65. Omitting the selection of predictors from the bootstrap procedure led to a severe underestimation of the optimism (decrease 0.006). The standard error of the observed ROC area in the independent validation samples was large (0.05). We recommend bootstrapping for internal validation because it gives reasonably valid estimates of the expected optimism in predictive performance provided that any selection of predictors is taken into account. For external validation, substantial sample sizes should be used for sufficient power to detect clinically important changes in performance as compared with the internally validated estimate.