External Validation Of Prediction Model Research Articles

Multicenter external validation of prediction models for clinical outcomes after spinal fusion for lumbar degenerative disease.

Clinical prediction models (CPM), such as the SCOAP-CERTAIN tool, can be utilized to enhance decision-making for lumbar spinal fusion surgery by providing quantitative estimates of outcomes, aiding surgeons in assessing potential benefits and risks for each individual patient. External validation is crucial in CPM to assess generalizability beyond the initial dataset. This ensures performance in diverse populations, reliability and real-world applicability of the results. Therefore, we externally validated the tool for predictability of improvement in oswestry disability index (ODI), back and leg pain (BP, LP). Prospective and retrospective data from multicenter registry was obtained. As outcome measure minimum clinically important change was chosen for ODI with ≥ 15-point and ≥ 2-point reduction for numeric rating scales (NRS) for BP and LP 12months after lumbar fusion for degenerative disease. We externally validate this tool by calculating discrimination and calibration metrics such as intercept, slope, Brier Score, expected/observed ratio, Hosmer-Lemeshow (HL), AUC, sensitivity and specificity. We included 1115 patients, average age 60.8 ± 12.5years. For 12-month ODI, area-under-the-curve (AUC) was 0.70, the calibration intercept and slope were 1.01 and 0.84, respectively. For NRS BP, AUC was 0.72, with calibration intercept of 0.97 and slope of 0.87. For NRS LP, AUC was 0.70, with calibration intercept of 0.04 and slope of 0.72. Sensitivity ranged from 0.63 to 0.96, while specificity ranged from 0.15 to 0.68. Lack of fit was found for all three models based on HL testing. Utilizing data from a multinational registry, we externally validate the SCOAP-CERTAIN prediction tool. The model demonstrated fair discrimination and calibration of predicted probabilities, necessitating caution in applying it in clinical practice. We suggest that future CPMs focus on predicting longer-term prognosis for this patient population, emphasizing the significance of robust calibration and thorough reporting.

External validation of predictive models for new vertebral fractures following percutaneous vertebroplasty.

To investigate the external validation and scalability of fourpredictive models regarding new vertebral fractures following percutaneous vertebroplasty. Utilizing retrospective data acquired from two centers, compute the area under the curve (AUC), calibration curve, and Kaplan-Meier plot to assess the model's discrimination and calibration. In the external validation of Zhong et al.'s 2015 predictive model for the probability of new fractures post-vertebroplasty, the AUC for re-fracture at 1, 2, and 3years postoperatively was 0.570, 0.617, and 0.664, respectively. The AUC for Zhong et al.'s 2016 predictive model for the probability of new fractures in neighboring vertebrae was 0.738. Kaplan-Meier plot results for both models indicated a significantly lower incidence of re-fracture in low-risk patients compared to high-risk patients. Li et al.'s 2021 model had an AUC of 0.518, and its calibration curve suggested an overestimation of the probability of new fractures. Li et al.'s 2022 model had an AUC of 0.556, and its calibration curve suggested an underestimation of the probability of new fractures. The external validation of four models demonstrated that the predictive model proposed by Zhong et al. in 2016 exhibited superior external generalization capabilities.

External Validation of Predictive Models for Failed Medical Management of Spinal Epidural Abscess

ObjectiveThere is limited consensus regarding management of spinal epidural abscesses (SEA), particularly in patients without neurologic deficits. Several models have been created to predict failure of medical management in patients with SEA. We evaluate the external validity of five predictive models in an independent cohort of patients with SEA. Methods176 patients with SEA between 2010 and 2019 at our institution were identified, and variables relevant to each predictive model were collected. Published prediction models were used to assign probability of medical management failure to each patient. Predicted probabilities of medical failure and actual patient outcomes were used to create Receiver Operating Characteristic (ROC) curves, with the area under the ROC curve (AUROC) used to quantify a model’s discriminative ability. Calibration curves were plotted using predicted probabilities and actual outcomes. The Spiegelhalter Z-test was used to determine adequate model calibration. ResultsOne model (Kim et al.) demonstrated good discriminative ability and adequate model calibration in our cohort (ROC = 0.831, p-value = 0.83). Parameters included in the model were age >65, diabetes, MRSA infection, and neurologic impairment. Four additional models did not perform well for discrimination or calibration metrics (Patel et al., ROC=0.580, p=<0.0001; Shah et al., ROC=0.653, p=<0.0001; Baum et al., ROC=0.498, p=<0.0001; Page et al., ROC=0.534, p=<0.0001). ConclusionOnly one published predictive model demonstrated acceptable discrimination and calibration in our cohort, suggesting limited generalizability of the evaluated models. Multi-institutional data may facilitate the development of widely applicable models to predict medical management failure in patients with SEA.

Functional Outcomes and Symptomatic Intracranial Hemorrhage After Endovascular Treatment in Acute Vertebrobasilar Artery Occlusions: External Validation of Prediction Models

Background Vertebrobasilar artery occlusion (VBAO) is a severe type of stroke. Multiple prediction models for outcome and symptomatic intracranial hemorrhage (sICH) of patients with acute ischemic stroke treated with endovascular treatment have been developed to improve patient management, but few are based on VBAO. This study aimed to provide an overview of published models to predict functional outcome and sICH as well as to validate their ability in patients with acute VBAO treated with endovascular treatment. Methods We performed a systematic search to identify models either developed or validated to predict functional outcomes or sICH after endovascular treatment. Models were externally validated in the Posterior Circulation Ischemic Stroke Registry (PERSIST) study (n = 2422). Outcome measures included the modified Rankin Scale (mRS) score at 90 days and sICH. Model performance was evaluated with discrimination (c‐statistic) and calibration (slope and intercept). Results A total of 65 models were included in overview. The most frequently used predictors were baseline National Institutes of Health Stroke Scale score (n = 57), age (n = 45), and glucose (n = 32). In the external validation cohort, 777 of 2353 patients (33.0%) achieved mRS score 0–2 at 90 days, 1061 of 2353 patients (45.1%) patients achieved mRS score 0–3 at 90 days, and sICH occurred in 170 of 2422 patients (7.0%). Finally, 27 models were included in external validation. For functional outcome models focusing on mRS score 0–2/3–6, discrimination ranged from 0.63 to 0.66 and best calibrated model was SC (Stroke Checkerboard) (intercept, −0.13 [95% CI, −0.27 to 0.01]; slope, 0.92 [95% CI, 0.67–1.17]). For functional outcome models focusing on mRS score 0–3/4–6, discrimination ranged from 0.64 to 0.74 and best calibrated model was modified Houston Intra‐Arterial Therapy 2 (mHIAT2) (intercept, −0.12 [95% CI, −0.31 to 0.07]; slope, 0.85 [95% CI, 0.65–1.04]). For sICH models, discrimination ranged from 0.53 to 0.83 and best calibrated model was Thrombolysis in Cerebral Infarction score, Alberta Stroke Program Early CT Score, and glucose (TAG) (intercept, 0.13 [95% CI, −0.25 to 0.51]; slope, 0.93 [95% CI, 0.63–1.23]). Conclusions The currently published models are inadequate for predicting functional outcomes and sICH in patients with acute VBAO undergoing endovascular treatment and, therefore, there is a need for more effective models specifically developed for VBAO conditions.

External validation of predictive models of sexual, urinary, bowel and hormonal function after surgery in prostate cancer subjects

BackgroundIn 2020, a research group published five linear longitudinal models, predict Expanded Prostate Cancer Index Composite-26 (EPIC-26) scores post-treatment for radical prostatectomy, external beam radiotherapy and active surveillance collectively in US patients with localized prostate cancer.MethodsOur study externally validates the five prediction models for patient reported outcomes post-surgery for localised prostate cancer. The models’ calibration, fit, variance explained and discrimination (concordance-indices) were assessed. Two Australian validation cohorts 1 and 2 years post-prostatectomy were constructed, consisting of 669 and 439 subjects, respectively (750 in total). Patient reported function in five domains post-prostatectomy: sexual, bowel, hormonal, urinary incontinence and other urinary dysfunction (irritation/obstruction). Domain function was assessed using the EPIC-26 questionnaire.Results1 year post-surgery, R2 was highest for the sexual domain (35%, SD = 0.02), lower for the bowel (21%, SD = 0.03) and hormone (15%, SD = 0.03) domains, and close to zero for urinary incontinence (1%, SD = 0.01) and irritation/obstruction (− 5%, SD = 0.04). Calibration slopes for these five models were 1.04 (SD = 0.04), 0.84 (SD = 0.06), 0.85 (SD = 0.06), 1.16 (SD = 0.13) and 0.45 (SD = 0.04), respectively. Calibration-in-the-large values were − 2.2 (SD = 0.6), 2.1 (SD = 0.01), 5.1 (SD = 0.1), 9.6 (SD = 0.9) and 4.0 (SD = 0.2), respectively. Concordance-indices were 0.73, 0.70, 0.70, 0.58 and 0.62, respectively (all had SD = 0.01). Mean absolute error and root mean square error were similar across the validation and development cohorts. The validation measures were largely similar at 2 years post-surgery.ConclusionsThe sexual, bowel and hormone domain models validated well and show promise for accurately predicting patient reported outcomes in a non-US surgical population. The urinary domain models validated poorly and may require recalibration or revision.

External validation of prediction models for vaginal delivery after the trial of labour among women with previous one caesarean section – A cohort study

ObjectiveTo externally validate three predictive models (the Grobman model (2007), the Zhang model (2020), and the Grobman model (2021)) for identifying women with increased chances of a successful trial of labour after caesarean section (TOLAC). MethodsThis retrospective observational cohort study was conducted in a tertiary teaching hospital from 2018 to 2021. Individual probabilities were calculated for women with previous one caesarean section who underwent TOLAC at term, using the predicted probabilities from the logistic regression models. The primary outcome of this study was vaginal delivery following attempted TOLAC. The predictive ability of the models was assessed using the area under the receiver operative characteristics curves (AUC) and a calibration graph. ResultsOf 1515 eligible women who underwent TOLAC, we found an overall rate of successful TOLAC of 60.3 %. No significant difference was noticed in adverse scar outcome and neonatal morbidity while comparing successful and failed TOLAC. The discriminative ability of Grobman-2007 and Grobman-2021 and the Zhang model were fair to poor with the AUC of 0.54(95 % CI 0.51–0.57), 0.62(95 % CI 0.59–0.65) and 0.66(95 % CI 0.63–0.69) respectively. The agreement between the observed rates of TOLAC success and the predicted probabilities for all three models was poor. ConclusionThe performance of all three models predicting success after TOLAC was poor in the study population. A population-specific model may be needed, with the addition of factors influencing the labour, such as the methods of induction, which may aid in predicting the outcome.

Development and external validation of prediction models for critical outcomes of unvaccinated COVID-19 patients based on demographics, medical conditions and dental status

BackgroundMultiple prediction models were developed for critical outcomes of COVID-19. However, prediction models using predictors which can be easily obtained in clinical practice and on dental status are scarce. AimThe study aimed to develop and externally validate prediction models for critical outcomes of COVID-19 for unvaccinated adult patients in hospital settings based on demographics, medical conditions, and dental status. MethodsA total of 285 and 352 patients from two hospitals in the Netherlands were retrospectively included as derivation and validation cohorts. Demographics, medical conditions, and dental status were considered potential predictors. The critical outcomes (death and ICU admission) were considered endpoints. Logistic regression analyses were used to develop two models: for death alone and for critical outcomes. The performance and clinical values of the models were determined in both cohorts. ResultsAge, number of teeth, chronic kidney disease, hypertension, diabetes, and chronic obstructive pulmonary diseases were the significant independent predictors. The models showed good to excellent calibration with observed: expected (O:E) ratios of 0.98 (95%CI: 0.76 to 1.25) and 1.00 (95%CI: 0.80 to 1.24), and discrimination with shrunken area under the curve (AUC) values of 0.85 and 0.79, based on the derivation cohort. In the validation cohort, the models showed good to excellent discrimination with AUC values of 0.85 (95%CI: 0.80 to 0.90) and 0.78 (95%CI: 0.73 to 0.83), but an overestimation in calibration with O:E ratios of 0.65 (95%CI: 0.49 to 0.85) and 0.67 (95%CI: 0.52 to 0.84). ConclusionThe performance of the models was acceptable in both derivation and validation cohorts. Number of teeth was an additive important predictor of critical outcomes of COVID-19. It is an easy-to-apply tool in hospitals for risk stratification of COVID-19 prognosis.

Sex and race/ethnicity specific reference predictive equations for abdominal adiposity indices using anthropometry in US adults

Background and aimsAbdominal adiposity indices have stronger associations with cardiometabolic risk factors compared to anthropometric measures but are rarely used in large scale studies due to the cost and efficiency. The aim of this study is to establish sex and race/ethnicity specific reference equations using anthropometric measures. Methods and resultsA secondary data analysis (n = 6589) of healthy adults was conducted using data from National Health and Nutrition Examination Survey 2011–2018. Variables included in the analyses were anthropometric measures (height; weight; waist circumference, WC) and abdominal adiposity indices (android percent fat; android to gynoid ratio, A/G ratio; visceral adipose tissue area, VATA; visceral to subcutaneous adipose area ratio, VSR). Multivariable prediction models were developed using quantile regression. Bland–Altman was used for external validation of prediction models. Reference equations to estimate android percent fat, A/G ratio, VATA and VSR from anthropometric measurements were developed using a randomly selected subsample of 4613. These reference equations for four abdominal adiposity indices were then cross-validated in the remaining subsample of 1976. The measured and predicted android percent fat, A/G ratio, VATA and VSR were not statistically different (p > 0.05) except for the A/G ratio in Asian males and VSR in White females. The results of Bland–Altman further revealed that ≥93% of predicted abdominal adiposity indices fell within the limits of agreement (±1.96 standard deviation). ConclusionThe sex and race/ethnicity specific reference equations for abdominal adiposity indices established using anthropometrics in the present study have strong predictive ability in US healthy adults.

Clinical Prediction Models for Recurrence in Patients with Resectable Grade 1 and 2 Sporadic Non-Functional Pancreatic Neuroendocrine Tumors: A Systematic Review.

Recurrence after resection in patients with non-functional pancreatic neuroendocrine tumors (NF-pNET) has a considerable impact on overall survival. Accurate risk stratification will tailor optimal follow-up strategies. This systematic review assessed available prediction models, including their quality. This systematic review followed PRISMA and CHARMS guidelines. PubMed, Embase, and the Cochrane Library were searched up to December 2022 for studies that developed, updated, or validated prediction models for recurrence in resectable grade 1 or 2 NF-pNET. Studies were critically appraised. After screening 1883 studies, 14 studies with 3583 patients were included: 13 original prediction models and 1 prediction model validation. Four models were developed for preoperative and nine for postoperative use. Six models were presented as scoring systems, five as nomograms, and two as staging systems. The c statistic ranged from 0.67 to 0.94. The most frequently included predictors were tumor grade, tumor size, and lymph node positivity. Critical appraisal deemed all development studies as having a high risk of bias and the validation study as having a low risk of bias. This systematic review identified 13 prediction models for recurrence in resectable NF-pNET with external validations for 3 of them. External validation of prediction models improves their reliability and stimulates use in daily practice.

External Validation of Prediction Models for Surgical Complications in People Considering Total Hip or Knee Arthroplasty Was Successful for Delirium but Not for Surgical Site Infection, Postoperative Bleeding, and Nerve Damage: A Retrospective Cohort Study

Although several models for the prediction of surgical complications after primary total hip or total knee replacement (THA and TKA, respectively) are available, only a few models have been externally validated. The aim of this study was to externally validate four previously developed models for the prediction of surgical complications in people considering primary THA or TKA. We included 2614 patients who underwent primary THA or TKA in secondary care between 2017 and 2020. Individual predicted probabilities of the risk for surgical complication per outcome (i.e., surgical site infection, postoperative bleeding, delirium, and nerve damage) were calculated for each model. The discriminative performance of patients with and without the outcome was assessed with the area under the receiver operating characteristic curve (AUC), and predictive performance was assessed with calibration plots. The predicted risk for all models varied between <0.01 and 33.5%. Good discriminative performance was found for the model for delirium with an AUC of 84% (95% CI of 0.82-0.87). For all other outcomes, poor discriminative performance was found; 55% (95% CI of 0.52-0.58) for the model for surgical site infection, 61% (95% CI of 0.59-0.64) for the model for postoperative bleeding, and 57% (95% CI of 0.53-0.61) for the model for nerve damage. Calibration of the model for delirium was moderate, resulting in an underestimation of the actual probability between 2 and 6%, and exceeding 8%. Calibration of all other models was poor. Our external validation of four internally validated prediction models for surgical complications after THA and TKA demonstrated a lack of predictive accuracy when applied in another Dutch hospital population, with the exception of the model for delirium. This model included age, the presence of a heart disease, and the presence of a disease of the central nervous system as predictor variables. We recommend that clinicians use this simple and straightforward delirium model during preoperative counselling, shared decision-making, and early delirium precautionary interventions.

External validation of prediction models for patient-reported outcome measurements collected using the selfBACK mobile app

BackgroundExternal validation is essential in examining the disparities in the training and validation cohorts during the development of prediction models, especially when the application domain is healthcare-oriented. Currently, the use of prediction models in healthcare research aimed at utilising the under-explored potential of patient-reported outcome measurements (PROMs) is limited, and few are validated using external datasets. ObjectiveTo validate the machine learning prediction models developed in our previous work [29] for predicting four pain-related patient-reported outcomes from the selfBACK clinical trial datasets. MethodsWe evaluate the validity of three pre-trained prediction models based on three methods— Case-Based Reasoning, Support Vector Regression, and XGBoost Regression—using an external dataset that contains PROMs collected from patients with non-specific neck and or low back pain using the selfBACK mobile application. ResultsOverall, the predictive power was low, except for prediction of one of the outcomes. The results indicate that while the predictions are far from immaculate in either case, the models show ability to generalise and predict outcomes for a new dataset. ConclusionExternal validation of the prediction models presents modest results and highlights the individual differences and need for external validation of prediction models in clinical settings. There is need for further development in this area of machine learning application and patient-centred care.

Development and external validation of prediction models for adverse health outcomes in rheumatoid arthritis: A multinational real-world cohort analysis

BackgroundIdentification of rheumatoid arthritis (RA) patients at high risk of adverse health outcomes remains a major challenge. We aimed to develop and validate prediction models for a variety of adverse health outcomes in RA patients initiating first-line methotrexate (MTX) monotherapy. MethodsData from 15 claims and electronic health record databases across 9 countries were used. Models were developed and internally validated on Optum® De-identified Clinformatics® Data Mart Database using L1-regularized logistic regression to estimate the risk of adverse health outcomes within 3 months (leukopenia, pancytopenia, infection), 2 years (myocardial infarction (MI) and stroke), and 5 years (cancers [colorectal, breast, uterine] after treatment initiation. Candidate predictors included demographic variables and past medical history. Models were externally validated on all other databases. Performance was assessed using the area under the receiver operator characteristic curve (AUC) and calibration plots. FindingsModels were developed and internally validated on 21,547 RA patients and externally validated on 131,928 RA patients. Models for serious infection (AUC: internal 0.74, external ranging from 0.62 to 0.83), MI (AUC: internal 0.76, external ranging from 0.56 to 0.82), and stroke (AUC: internal 0.77, external ranging from 0.63 to 0.95), showed good discrimination and adequate calibration. Models for the other outcomes showed modest internal discrimination (AUC < 0.65) and were not externally validated. InterpretationWe developed and validated prediction models for a variety of adverse health outcomes in RA patients initiating first-line MTX monotherapy. Final models for serious infection, MI, and stroke demonstrated good performance across multiple databases and can be studied for clinical use. FundingThis activity under the European Health Data & Evidence Network (EHDEN) has received funding from the Innovative Medicines Initiative 2 Joint Undertaking under grant agreement No 806968. This Joint Undertaking receives support from the European Union's Horizon 2020 research and innovation programme and EFPIA.

Predicting the earliest deviation in weight gain in the course towards manifest overweight in offspring exposed to obesity in pregnancy: a longitudinal cohort study

BackgroundObesity in pregnancy and related early-life factors place the offspring at the highest risk of being overweight. Despite convincing evidence on these associations, there is an unmet public health need to identify “high-risk” offspring by predicting very early deviations in weight gain patterns as a subclinical stage towards overweight. However, data and methods for individual risk prediction are lacking. We aimed to identify those infants exposed to obesity in pregnancy at ages 3 months, 1 year, and 2 years who likely will follow a higher-than-normal body mass index (BMI) growth trajectory towards manifest overweight by developing an early-risk quantification system.MethodsThis study uses data from the prospective mother-child cohort study Programming of Enhanced Adiposity Risk in CHildhood–Early Screening (PEACHES) comprising 1671 mothers with pre-conception obesity and without (controls) and their offspring. Exposures were pre- and postnatal risks documented in patient-held maternal and child health records. The main outcome was a “higher-than-normal BMI growth pattern” preceding overweight, defined as BMI z-score >1 SD (i.e., World Health Organization [WHO] cut-off “at risk of overweight”) at least twice during consecutive offspring growth periods between age 6 months and 5 years. The independent cohort PErinatal Prevention of Obesity (PEPO) comprising 11,730 mother-child pairs recruited close to school entry (around age 6 years) was available for data validation. Cluster analysis and sequential prediction modelling were performed.ResultsData of 1557 PEACHES mother-child pairs and the validation cohort were analyzed comprising more than 50,000 offspring BMI measurements. More than 1-in-5 offspring exposed to obesity in pregnancy belonged to an upper BMI z-score cluster as a distinct pattern of BMI development (above the cut-off of 1 SD) from the first months of life onwards resulting in preschool overweight/obesity (age 5 years: odds ratio [OR] 16.13; 95% confidence interval [CI] 9.98–26.05). Contributing early-life factors including excessive weight gain (OR 2.08; 95% CI 1.25–3.45) and smoking (OR 1.94; 95% CI 1.27–2.95) in pregnancy were instrumental in predicting a “higher-than-normal BMI growth pattern” at age 3 months and re-evaluating the risk at ages 1 year and 2 years (area under the receiver operating characteristic [AUROC] 0.69–0.79, sensitivity 70.7–76.0%, specificity 64.7–78.1%). External validation of prediction models demonstrated adequate predictive performances.ConclusionsWe devised a novel sequential strategy of individual prediction and re-evaluation of a higher-than-normal weight gain in “high-risk” infants well before developing overweight to guide decision-making. The strategy holds promise to elaborate interventions in an early preventive manner for integration in systems of well-child care.

Prospective Multicenter Observational Cohort Study on Time to Death in Potential Controlled Donation After Circulatory Death Donors-Development and External Validation of Prediction Models: The DCD III Study.

Acceptance of organs from controlled donation after circulatory death (cDCD) donors depends on the time to circulatory death. Here we aimed to develop and externally validate prediction models for circulatory death within 1 or 2 h after withdrawal of life-sustaining treatment. In a multicenter, observational, prospective cohort study, we enrolled 409 potential cDCD donors. For model development, we applied the least absolute shrinkage and selection operator (LASSO) regression and machine learning-artificial intelligence analyses. Our LASSO models were validated using a previously published cDCD cohort. Additionally, we validated 3 existing prediction models using our data set. For death within 1 and 2 h, the area under the curves (AUCs) of the LASSO models were 0.77 and 0.79, respectively, whereas for the artificial intelligence models, these were 0.79 and 0.81, respectively. We were able to identify 4% to 16% of the patients who would not die within these time frames with 100% accuracy. External validation showed that the discrimination of our models was good (AUCs 0.80 and 0.82, respectively), but they were not able to identify a subgroup with certain death after 1 to 2 h. Using our cohort to validate 3 previously published models showed AUCs ranging between 0.63 and 0.74. Calibration demonstrated that the models over- and underestimated the predicted probability of death. Our models showed a reasonable ability to predict circulatory death. External validation of our and 3 existing models illustrated that their predictive ability remained relatively stable. We accurately predicted a subset of patients who died after 1 to 2 h, preventing starting unnecessary donation preparations, which, however, need external validation in a prospective cohort.

Identification of patients with ductal carcinoma in situ at high risk of postoperative upstaging: A comprehensive review and an external (un)validation of predictive models developed

BackgroundBetween 8% and 56% of pure ductal carcinoma in situ are upstaged to microinvasive or invasive carcinoma on definitive pathological examination. The first objective of this study was to perform a comprehensive review of the literature on factors associated with increased risk of pre-operative underestimation. The second objective was to perform an external validation of the predictive models developed to enable their use in daily practice if relevant. Materials and methodsA literature search using Medline was undertaken. For each model selected, external validation within the study cohort was undertaken. The study cohort consisted of patients with histologically proven ductal carcinoma in situ who underwent surgical treatment at a French referral centre for cancer treatment between January 2007 and November 2018. ResultsTwo hundred and thirteen articles were identified; of these, 34 articles focused on factors associated with pre-operative underestimation of invasive carcinoma, 11 studies were identified as predictive models, and three studies were selected for external validation within the study cohort. Four hundred and eighty-eight eligible patients were identified in the study cohort, with an underestimation rate of 9.2%. The Jakub nomogram concordance index was 0.45 [95% confidence interval (CI) 0.39–0.51], the Park nomogram concordance index was 0.57 (95% CI 0.48–0.55), and the Coufal nomogram concordance index was 0.52 (95% CI 0.48–0.55). ConclusionWhile the literature is rich on this topic, this review clearly highlights the lack of consensus regarding parameters associated with underestimation. It was not possible to validate previously published models for use in daily practice.

Minimum sample size calculations for external validation of a clinical prediction model with a time-to-event outcome.

Previous articles in Statistics in Medicine describe how to calculate the sample size required for external validation of prediction models with continuous and binary outcomes. The minimum sample size criteria aim to ensure precise estimation of key measures of a model's predictive performance, including measures of calibration, discrimination, and net benefit. Here, we extend the sample size guidance to prediction models with a time-to-event (survival) outcome, to cover external validation in datasets containing censoring. A simulation-based framework is proposed, which calculates the sample size required to target a particular confidence interval width for the calibration slope measuring the agreement between predicted risks (from the model) and observed risks (derived using pseudo-observations to account for censoring) on the log cumulative hazard scale. Precise estimation of calibration curves, discrimination, and net-benefit can also be checked in this framework. The process requires assumptions about the validation population in terms of the (i) distribution of the model's linear predictor and (ii) event and censoring distributions. Existing information can inform this; in particular, the linear predictor distribution can be approximated using the C-index or Royston's D statistic from the model development article, together with the overall event risk. We demonstrate how the approach can be used to calculate the sample size required to validate a prediction model for recurrent venous thromboembolism. Ideally the sample size should ensure precise calibration across the entire range of predicted risks, but must at least ensure adequate precision in regions important for clinical decision-making. Stata and R code are provided.

Effective Search of Triterpenes with Anti-HSV-1 Activity Using a Classification Model by Logistic Regression.

Natural products are an excellent source of skeletons for medicinal seeds. Triterpenes and saponins are representative natural products that exhibit anti-herpes simplex virus type 1 (HSV-1) activity. However, there has been a lack of comprehensive information on the anti-HSV-1 activity of triterpenes. Therefore, expanding information on the anti-HSV-1 activity of triterpenes and improving the efficiency of their exploration are urgently required. To improve the efficiency of the development of anti-HSV-1 active compounds, we constructed a predictive model for the anti-HSV-1 activity of triterpenes by using the information obtained from previous studies using machine learning methods. In this study, we constructed a binary classification model (i.e., active or inactive) using a logistic regression algorithm. As a result of the evaluation of predictive model, the accuracy for the test data is 0.79, and the area under the curve (AUC) is 0.86. Additionally, to enrich the information on the anti-HSV-1 activity of triterpenes, a plaque reduction assay was performed on 20 triterpenes. As a result, chikusetsusaponin IVa (11: IC50 = 13.06 μM) was found to have potent anti-HSV-1 with three potentially anti-HSV-1 active triterpenes. The assay result was further used for external validation of predictive model. The prediction of the test compounds in the activity test showed a high accuracy (0.83) and AUC (0.81). We also found that this predictive model was found to be able to successfully narrow down the active compounds. This study provides more information on the anti-HSV-1 activity of triterpenes. Moreover, the predictive model can improve the efficiency of the development of active triterpenes by integrating many previous studies to clarify potential relationships.

Risk of caesarean delivery in labour induction: a systematic review and external validation of predictive models.

Despite the existence of numerous published models predicting the risk of caesarean delivery in women undergoing induction of labour (IOL), validated models are scarce. To systematically review and externally assess the predictive capacity of caesarean delivery risk models in women undergoing IOL. Studies published up to 15 January 2021 were identified through PubMed, CINAHL, Scopus and ClinicalTrials.gov, without temporal or language restrictions. Studies describing the derivation of new models for predicting the risk of caesarean delivery in labour induction. Three authors independently screened the articles and assessed the risk of bias (ROB) according to the prediction model risk of bias assessment tool (PROBAST). External validation was performed in a prospective cohort of 468 pregnancies undergoing IOL from February 2019 to August 2020. The predictive capacity of the models was assessed by creating areas under the receiver operating characteristic curve (AUCs), calibration plots and decision curve analysis (DCA). Fifteen studies met the eligibility criteria; 12 predictive models were validated. The quality of most of the included studies was not adequate. The AUC of the models varied from 0.520 to 0.773. The three models with the best discriminative capacity were those of Levine et al. (AUC 0.773, 95% CI 0.720-0.827), Hernández et al. (AUC 0.762, 95% CI 0.715-0.809) and Rossi et al. (AUC 0.752, 95% CI 0.707-0.797). Predictive capacity and methodological quality were limited; therefore, we cannot currently recommend the use of any of the models for decision making in clinical practice. Predictive models that predict the risk of cesarean section in labor inductions are currently not applicable.

External validation of prediction models for obstructive coronary artery disease in patients with suspected stable coronary artery disease

Abstract Background Pre-test probability (PTP) is an important tool in the diagnostic work-up for obstructive coronary artery disease (CAD) but must be calibrated to the declining disease prevalence in patients referred to diagnostic testing. Purpose To externally validate the published basic and clinical PTP models in a contemporary angina cohort with low prevalence of CAD and to compare with the reference European Society of Cardiology 2019 PTP (ESC 2019 PTP). Methods The validation cohort consisted of 42.328 patients (54% women, age ≥30 years, no previous CAD) with symptoms of CAD referred for cardiac computed tomography angiography in the western region of Denmark from 2008–2017 (3.3 million inhabitants). Obstructive CAD was defined from either invasive angiography as stenosis &amp;gt;50%, or when performed, from FFR &amp;lt;0.8 in coronary arteries with diameters &amp;gt;2 mm. The basic prediction model included type of angina, sex, and age, and the clinical model added diabetes, family history of CAD, and dyslipidemia. The ESC 2019 PTP was calculated from age, sex, and angina symptoms. Discrimination, calibration, and negative predictive value (NPV) were measured for all three models. Results Obstructive CAD was present in 3718 (8.8%). In the ESC 2019 PTP model, the basic model, and the clinical model 19.5%, 48.5%, and 55.7% were classified as very low risk and only 1.6%, 3.7%, and 3.5% of these had obstructive CAD, respectively (figure 1). Discrimination was similar for the three models with AUC of 0.76 (95% CI 0.75–0.77), 0.74 [0.73–0.75], and 0.76 [0.75–0.76], for the ESC 2019 PTP, basic, and clinical model, respectively. At the clinically relevant very low predicted probability (≤5%) of CAD, the clinical and basic model were very well calibrated, whereas the ESC 2019 PTP model overestimated the CAD prevalence. NPV at cut-off ≤5% were 98.4% [98.1–98.7] for the ESC 2019 PTP model, 96.3% [96.1–96.6] for the basic model, and 96.5% [96.3–96.7] for the clinical model. At cut-off &amp;lt;15%, NPVs were 96.1% [95.8–96.3] for the ESC 2019 PTP model, 94.5% [94.2–94.7] for the basic model, and 94.2% [94.0–94.5] for the clinical model. Conclusion In a population with a prevalence of 8.8% obstructive CAD, a clinical prediction model including diabetes, family history of CAD, and dyslipidemia in addition to the variables of the ESC 2019 PTP model ruled out 36.2% more patients than the ESC 2019 PTP model (23.592 vs. 8245 patients) while only overlooking 1.9% more cases of obstructive CAD when choosing a cut-off ≤5%. Use of this model is therefore potentially cost saving. Funding Acknowledgement Type of funding sources: None. Figure 1Figure 2

Dementia risk in the general population: large-scale external validation of prediction models in the AGES-Reykjavik study

We aimed to evaluate the external performance of prediction models for all-cause dementia or AD in the general population, which can aid selection of high-risk individuals for clinical trials and prevention. We identified 17 out of 36 eligible published prognostic models for external validation in the population-based AGES-Reykjavik Study. Predictive performance was assessed with c statistics and calibration plots. All five models with a c statistic > .75 (.76–.81) contained cognitive testing as a predictor, while all models with lower c statistics (.67–.75) did not. Calibration ranged from good to poor across all models, including systematic risk overestimation or overestimation for particularly the highest risk group. Models that overestimate risk may be acceptable for exclusion purposes, but lack the ability to accurately identify individuals at higher dementia risk. Both updating existing models or developing new models aimed at identifying high-risk individuals, as well as more external validation studies of dementia prediction models are warranted.