Validation Of Multivariable Prediction Model Research Articles

Systematic review of methods used in prediction models with recurrent event data

BackgroundPatients who suffer from chronic conditions or diseases are susceptible to experiencing repeated events of the same type (e.g. seizures), termed ‘recurrent events’. Prediction models can be used to predict the risk of recurrence so that intervention or management can be tailored accordingly, but statistical methodology can vary. The objective of this systematic review was to identify and describe statistical approaches that have been applied for the development and validation of multivariable prediction models with recurrent event data. A secondary objective was to informally assess the characteristics and quality of analysis approaches used in the development and validation of prediction models of recurrent event data.MethodsSearches were run in MEDLINE using a search strategy in 2019 which included index terms and phrases related to recurrent events and prediction models. For studies to be included in the review they must have developed or validated a multivariable clinical prediction model for recurrent event outcome data, specifically modelling the recurrent events and the timing between them.The statistical analysis methods used to analyse the recurrent event data in the clinical prediction model were extracted to answer the primary aim of the systematic review. In addition, items such as the event rate as well as any discrimination and calibration statistics that were used to assess the model performance were extracted for the secondary aim of the review.ResultsA total of 855 publications were identified using the developed search strategy and 301 of these are included in our systematic review. The Andersen-Gill method was identified as the most commonly applied method in the analysis of recurrent events, which was used in 152 (50.5%) studies. This was closely followed by frailty models which were used in 116 (38.5%) included studies. Of the 301 included studies, only 75 (24.9%) internally validated their model(s) and three (1.0%) validated their model(s) in an external dataset.ConclusionsThis review identified a variety of methods which are used in practice when developing or validating prediction models for recurrent events. The variability of the approaches identified is cause for concern as it indicates possible immaturity in the field and highlights the need for more methodological research to bring greater consistency in approach of recurrent event analysis. Further work is required to ensure publications report all required information and use robust statistical methods for model development and validation.PROSPERO registrationCRD42019116031.

A systematic review of fall prediction models for community-dwelling older adults: comparison between models based on research cohorts and models based on routinely collected data.

Prediction models can identify fall-prone individuals. Prediction models can be based on either data from research cohorts (cohort-based) or routinely collected data (RCD-based). We review and compare cohort-based and RCD-based studies describing the development and/or validation of fall prediction models for community-dwelling older adults. Medline and Embase were searched via Ovid until January 2023. We included studies describing the development or validation of multivariable prediction models of falls in older adults (60+). Both risk of bias and reporting quality were assessed using the PROBAST and TRIPOD, respectively. We included and reviewed 28 relevant studies, describing 30 prediction models (23 cohort-based and 7 RCD-based), and external validation of two existing models (one cohort-based and one RCD-based). The median sample sizes for cohort-based and RCD-based studies were 1365 [interquartile range (IQR) 426-2766] versus 90441 (IQR 56442-128157), and the ranges of fall rates were 5.4% to 60.4% versus 1.6% to 13.1%, respectively. Discrimination performance was comparable between cohort-based and RCD-based models, with the respective area under the receiver operating characteristic curves ranging from 0.65 to 0.88 versus 0.71 to 0.81. The median number of predictors in cohort-based final models was 6 (IQR 5-11); for RCD-based models, it was 16 (IQR 11-26). All but one cohort-based model had high bias risks, primarily due to deficiencies in statistical analysis and outcome determination. Cohort-based models to predict falls in older adults in the community are plentiful. RCD-based models are yet in their infancy but provide comparable predictive performance with no additional data collection efforts. Future studies should focus on methodological and reporting quality.

Protocol for Development and Validation of Multivariable Prediction Models for Chronic Postsurgical Pain Following Video-Assisted Thoracic Surgery.

Chronic postsurgical pain (CPSP) is a common complication after thoracic surgery and associated with long-term adverse outcomes. This study aims to develop two prediction models for CPSP after video-assisted thoracic surgery (VATS). This single-center prospective cohort study will include a total of 500 adult patients undergoing VATS lung resection (n = 350 for development and n = 150 for external validation). Patients will be enrolled continuously at The First Affiliated Hospital of Soochow University in Suzhou, China. The cohort for external validation will be recruited in another time period. The outcome is CPSP, which is defined as pain with the numerical rating scale score of 1 or higher 3 months after VATS. Univariate and multivariable logistic regression analyses will be performed to develop two CPSP prediction models based on patients' data of postoperative day 1 and day 14, respectively. For internal validation, we will use the bootstrapping validation technique. For external validation, the discrimination capability of the models will be assessed using the area under the receiver operating characteristic curve, and the calibration will be evaluated using the calibration curve and Hosmer-Lemeshow goodness-of-fit statistic. The results will be presented in model formulas and nomograms. Based on the development and validation of the prediction models, our results contribute to early prediction and treatment of CPSP after VATS. Chinese Clinical Trial Register (ChiCTR2200066122).

Predicting patient-reported outcomes following lumbar spine surgery: development and external validation of multivariable prediction models

BackgroundThis study aimed to develop and externally validate prediction models of spinal surgery outcomes based on a retrospective review of a prospective clinical database, uniquely comparing multivariate regression and random forest (machine learning) approaches, and identifying the most important predictors.MethodsOutcomes were change in back and leg pain intensity and Core Outcome Measures Index (COMI) from baseline to the last available postoperative follow-up (3–24 months), defined as minimal clinically important change (MCID) and continuous change score. Eligible patients underwent lumbar spine surgery for degenerative pathology between 2011 and 2021. Data were split by surgery date into development (N = 2691) and validation (N = 1616) sets for temporal external validation. Multivariate logistic and linear regression, and random forest classification and regression models, were fit to the development data and validated on the external data.ResultsAll models demonstrated good calibration in the validation data. Discrimination ability (area under the curve) for MCID ranged from 0.63 (COMI) to 0.72 (back pain) in regression, and from 0.62 (COMI) to 0.68 (back pain) in random forests. The explained variation in continuous change scores spanned 16%-28% in linear, and 15%-25% in random forests regression. The most important predictors included age, baseline scores on the respective outcome measures, type of degenerative pathology, previous spinal surgeries, smoking status, morbidity, and duration of hospital stay.ConclusionsThe developed models appear robust and generalisable across different outcomes and modelling approaches but produced only borderline acceptable discrimination ability, suggesting the need to assess further prognostic factors. External validation showed no advantage of the random forest approach.

Stratified care in hip arthroscopy: can we predict successful and unsuccessful outcomes? Development and external temporal validation of multivariable prediction models

ObjectiveAlthough hip arthroscopy is a widely adopted treatment option for hip-related pain, it is unknown whether preoperative clinical information can be used to assist surgical decision-making to avoid offering surgery...

Development and validation of multivariable prediction models for in-hospital death, 30-day death, and change in residence after hip fracture surgery and the 'stratify-hip' algorithm.

To develop and validate the stratify-hip algorithm (multivariable prediction models to predict those at low, medium, and high risk across in-hospital death, 30-day death, and residence change after hip fracture). Multivariable Fine-Gray and logistic regression of audit data linked to hospital records for older adults surgically-treated for hip fracture in England/Wales 2011-2014 (development n=170,411) and 2015-2016 (external validation, n=90,102). Outcomes included time to in-hospital death, death at 30-days, and time to residence change. Predictors included: age, sex, pre-fracture mobility, dementia, and pre-fracture residence (not for residence change). Model assumptions, performance, and sensitivity to missingness were assessed. Models were incorporated into the stratify-hip algorithm assigning patients to overall low (low risk across outcomes), medium (low death risk, medium/high risk of residence change), or high (high risk of in-hospital death, high/medium risk of 30-day death) risk. For complete-case analysis, 6,780 of 141,158 patients (4.8%) died in-hospital, 8,693 of 149,258 patients (5.8%) died by 30-days, and 4,461 of 119,420 patients (3.7 %) had residence change. Models demonstrated acceptable calibration (observed:expected ratio 0.90, 0.99, and 0.94), and discrimination (area under curve 73.1, 71.1 and 71.5; Brier score 5.7, 5.3, 5.6) for in-hospital death, 30-day death, and residence change, respectively. Overall, 31%, 28%, and 41% of patients were assigned to overall low-, medium-, and high- risk. External validation and missing data analyses elicited similar findings. The algorithm is available at https://stratifyhip.co.uk. The current study developed and validated the stratify-hip algorithm as a new tool to risk stratify patients after hip fracture.

Development and Validation of Multivariable Prediction Models of Serological Response to SARS-CoV-2 Vaccination in Kidney Transplant Recipients

Development and Validation of Multivariable Prediction Models of Serological Response to SARS-CoV-2 Vaccination in Kidney Transplant Recipients

Development and validation of multivariable prediction models of serological response to SARS-CoV-2 vaccination in kidney transplant recipients.

Repeated vaccination against SARS-CoV-2 increases serological response in kidney transplant recipients (KTR) with high interindividual variability. No decision support tool exists to predict SARS-CoV-2 vaccination response to third or fourth vaccination in KTR. We developed, internally and externally validated five different multivariable prediction models of serological response after the third and fourth vaccine dose against SARS-CoV-2 in previously seronegative, COVID-19-naïve KTR. Using 20 candidate predictor variables, we applied statistical and machine learning approaches including logistic regression (LR), least absolute shrinkage and selection operator (LASSO)-regularized LR, random forest, and gradient boosted regression trees. For development and internal validation, data from 590 vaccinations were used. External validation was performed in four independent, international validation cohorts comprising 191, 184, 254, and 323 vaccinations, respectively. LASSO-regularized LR performed on the whole development dataset yielded a 20- and 10-variable model, respectively. External validation showed AUC-ROC of 0.840, 0.741, 0.816, and 0.783 for the sparser 10-variable model, yielding an overall performance 0.812. A 10-variable LASSO-regularized LR model predicts vaccination response in KTR with good overall accuracy. Implemented as an online tool, it can guide decisions whether to modulate immunosuppressive therapy before additional active vaccination, or to perform passive immunization to improve protection against COVID-19 in previously seronegative, COVID-19-naïve KTR.

Development and validation of multivariable prediction models for adverse COVID-19 outcomes in patients with IBD

ObjectivesDevelop an individualised prognostic risk prediction tool for predicting the probability of adverse COVID-19 outcomes in patients with inflammatory bowel disease (IBD).Design and settingThis study developed and validated prognostic penalised...

Statistical Development and Validation of Clinical Prediction Models.

Clinical prediction models in anesthesia and surgery research have many clinical applications including preoperative risk stratification with implications for clinical utility in decision-making, resource utilization, and costs. It is imperative that predictive algorithms and multivariable models are validated in a suitable and comprehensive way in order to establish the robustness of the model in terms of accuracy, predictive ability, reliability, and generalizability. The purpose of this article is to educate anesthesia researchers at an introductory level on important statistical concepts involved with development and validation of multivariable prediction models for a binary outcome. Methods covered include assessments of discrimination and calibration through internal and external validation. An anesthesia research publication is examined to illustrate the process and presentation of multivariable prediction model development and validation for a binary outcome. Properly assessing the statistical and clinical validity of a multivariable prediction model is essential for reassuring the generalizability and reproducibility of the published tool.

Development and internal validation of multivariable prediction models for biochemical failure after MRI-guided focal salvage high-dose-rate brachytherapy for radiorecurrent prostate cancer

Background and purposeMagnetic resonance-guided focal salvage high-dose-rate brachytherapy (FS-HDR-BT) for radiorecurrent prostate cancer (PCa) shows low toxicity rates. However, biochemical failure (BF) after treatment occurs frequently. We developed two prediction models for BF (Phoenix definition) with the aim of enhancing patient counselling before FS-HDR-BT and during follow-up. Materials and methodsA prospective cohort of 150 radiorecurrent PCa patients treated with FS-HDR-BT between 2013 and 2020 was used for model development and internal validation. Multivariable Cox Proportional Hazards regression was applied. For model 1, only pre-salvage variables were included as candidate predictors. For model 2, additional (post-)salvage characteristics were tested. After calibration, nomograms and webtools were constructed. Finally, three risk groups were identified. ResultsSixty-one patients (41%) experienced BF. At baseline (model 1), age, gross tumour volume, pre-salvage PSA, and pre-salvage PSA doubling time (PSADT) were predictive of BF. During follow-up (model 2), age, pre-salvage PSA and PSADT, seminal vesicle involvement, post-salvage time to PSA nadir, and percentage PSA reduction were predictive of BF. The adjusted C-statistics were 0.73 (95% CI: 0.66–0.81) and 0.84 (95% CI: 0.78–0.90), respectively, with acceptable calibration. Estimated 2-year biochemical disease-free survival for the low-, intermediate-, and high-risk groups were 84%, 70%, and 31% (model 1), and 100%, 71%, and 5% (model 2). ConclusionTwo models are provided for prediction of BF in patients with radiorecurrent PCa treated with FS-HDR-BT. Based on pre- and post-salvage characteristics, we are able to identify patients with a high risk of BF. These findings can aid patient counselling for FS-HDR-BT.

Development and validation of multivariable prediction models for adverse COVID-19 outcomes in patients with IBD

Development and validation of multivariable prediction models for adverse COVID-19 outcomes in patients with IBD

Validation of a Machine Learning Model to Predict Childhood Lead Poisoning

Childhood lead poisoning causes irreversible neurobehavioral deficits, but current practice is secondary prevention. To validate a machine learning (random forest) prediction model of elevated blood lead levels (EBLLs) by comparison with a parsimonious logistic regression. This prognostic study for temporal validation of multivariable prediction models used data from the Women, Infants, and Children (WIC) program of the Chicago Department of Public Health. Participants included a development cohort of children born from January 1, 2007, to December 31, 2012, and a validation WIC cohort born from January 1 to December 31, 2013. Blood lead levels were measured until December 31, 2018. Data were analyzed from January 1 to October 31, 2019. Blood lead level test results; lead investigation findings; housing characteristics, permits, and violations; and demographic variables. Incident EBLL (≥6 μg/dL). Models were assessed using the area under the receiver operating characteristic curve (AUC) and confusion matrix metrics (positive predictive value, sensitivity, and specificity) at various thresholds. Among 6812 children in the WIC validation cohort, 3451 (50.7%) were female, 3057 (44.9%) were Hispanic, 2804 (41.2%) were non-Hispanic Black, 458 (6.7%) were non-Hispanic White, and 442 (6.5%) were Asian (mean [SD] age, 5.5 [0.3] years). The median year of housing construction was 1919 (interquartile range, 1903-1948). Random forest AUC was 0.69 compared with 0.64 for logistic regression (difference, 0.05; 95% CI, 0.02-0.08). When predicting the 5% of children at highest risk to have EBLLs, random forest and logistic regression models had positive predictive values of 15.5% and 7.8%, respectively (difference, 7.7%; 95% CI, 3.7%-11.3%), sensitivity of 16.2% and 8.1%, respectively (difference, 8.1%; 95% CI, 3.9%-11.7%), and specificity of 95.5% and 95.1% (difference, 0.4%; 95% CI, 0.0%-0.7%). The machine learning model outperformed regression in predicting childhood lead poisoning, especially in identifying children at highest risk. Such a model could be used to target the allocation of lead poisoning prevention resources to these children.

Research round-up

In a US case-control study, Frye and colleagues investigated whether the occurrence of antibodies to infectious agents differs between individuals with bipolar disorder (n=1207) and their unaffected counterparts (n=745). Researchers categorised patients as cytomegalovirus (CMV) IgG positive (increased long-term antibody response) and Toxoplasma gondii IgG negative (decreased long-term antibody response) or CMV IgG negative and T gondii IgG positive. Cases with bipolar disorder had higher odds of being CMV IgG positive and T gondii IgG negative than control participants (odds ratio 1·33 [95% CI 1·09–1·62], p=0·004). C-reactive protein concentrations did not differ significantly between participants with bipolar disorder and control participants (p=0·73). Development and validation of multivariable prediction models of remission, recovery, and quality of life outcomes in people with first episode psychosis: a machine learning approachIn our machine learning analysis, we showed that prediction models can reliably and prospectively identify poor remission and recovery outcomes at 1 year for patients with first-episode psychosis using baseline clinical variables at first clinical contact. Full-Text PDF Open AccessCognitive behavioural therapy for irritable bowel syndrome: 24-month follow-up of participants in the ACTIB randomised trialAt 24-month follow-up, sustained improvements in IBS were seen in both CBT groups compared with TAU, although some previous gains were reduced compared with the 12-month outcomes. IBS-specific CBT has the potential to provide long-term improvement in IBS, achievable within a usual clinical setting. Increasing access to CBT for IBS could achieve long-term patient benefit. Full-Text PDF Open Access

Toward clinically useful models for individualised prognostication in psychosis

Toward clinically useful models for individualised prognostication in psychosis

Development and validation of multivariable prediction models of remission, recovery, and quality of life outcomes in people with first episode psychosis: a machine learning approach

Outcomes for people with first-episode psychosis are highly heterogeneous. Few reliable validated methods are available to predict the outcome for individual patients in the first clinical contact. In this study, we aimed to build multivariable prediction models of 1-year remission and recovery outcomes using baseline clinical variables in people with first-episode psychosis. In this machine learning approach, we applied supervised machine learning, using regularised regression and nested leave-one-site-out cross-validation, to baseline clinical data from the English Evaluating the Development and Impact of Early Intervention Services (EDEN) study (n=1027), to develop and internally validate prediction models at 1-year follow-up. We assessed four binary outcomes that were recorded at 1 year: symptom remission, social recovery, vocational recovery, and quality of life (QoL). We externally validated the prediction models by selecting from the top predictor variables identified in the internal validation models the variables shared with the external validation datasets comprised of two Scottish longitudinal cohort studies (n=162) and the OPUS trial, a randomised controlled trial of specialised assertive intervention versus standard treatment (n=578). The performance of prediction models was robust for the four 1-year outcomes of symptom remission (area under the receiver operating characteristic curve [AUC] 0·703, 95% CI 0·664-0·742), social recovery (0·731, 0·697-0·765), vocational recovery (0·736, 0·702-0·771), and QoL (0·704, 0·667-0·742; p<0·0001 for all outcomes), on internal validation. We externally validated the outcomes of symptom remission (AUC 0·680, 95% CI 0·587-0·773), vocational recovery (0·867, 0·805-0·930), and QoL (0·679, 0·522-0·836) in the Scottish datasets, and symptom remission (0·616, 0·553-0·679), social recovery (0·573, 0·504-0·643), vocational recovery (0·660, 0·610-0·710), and QoL (0·556, 0·481-0·631) in the OPUS dataset. In our machine learning analysis, we showed that prediction models can reliably and prospectively identify poor remission and recovery outcomes at 1 year for patients with first-episode psychosis using baseline clinical variables at first clinical contact. Lundbeck Foundation.

Prediction of remission in pharmacotherapy of untreated major depression: development and validation of multivariable prediction models.

Depression is increasingly recognized as a chronic and relapsing disorder. However, an important minority of patients who start treatment for their major depressive episode recover to euthymia. It is clinically important to be able to predict such individuals. The study is a secondary analysis of a recently completed pragmatic megatrial examining first- and second-line treatments for hitherto untreated episodes of non-psychotic unipolar major depression (n = 2011). Using the first half of the cohort as the derivation set, we applied multiply-imputed stepwise logistic regression with backward selection to build a prediction model to predict remission, defined as scoring 4 or less on the Patient Health Quetionnaire-9 at week 9. We used three successively richer sets of predictors at baseline only, up to week 1, and up to week 3. We examined the external validity of the derived prediction models with the second half of the cohort. In total, 37.0% (95% confidence interval 34.8-39.1%) were in remission at week 9. Only the models using data up to week 1 or 3 showed reasonable performance. Age, education, length of episode and depression severity remained in the multivariable prediction models. In the validation set, the discrimination of the prediction model was satisfactory with the area under the curve of 0.73 (0.70-0.77) and 0.82 (0.79-0.85), while the calibration was excellent with non-significant goodness-of-fit χ2 values (p = 0.41 and p = 0.29), respectively. Patients and clinicians can use these prediction models to estimate their predicted probability of achieving remission after acute antidepressant therapy.

External validation, update and development of prediction models for pre-eclampsia using an Individual Participant Data (IPD) meta-analysis: the International Prediction of Pregnancy Complication Network (IPPIC pre-eclampsia) protocol

BackgroundPre-eclampsia, a condition with raised blood pressure and proteinuria is associated with an increased risk of maternal and offspring mortality and morbidity. Early identification of mothers at risk is needed to target management.Methods/designWe aim to systematically review the existing literature to identify prediction models for pre-eclampsia. We have established the International Prediction of Pregnancy Complication Network (IPPIC), made up of 72 researchers from 21 countries who have carried out relevant primary studies or have access to existing registry databases, and collectively possess data from more than two million patients. We will use the individual participant data (IPD) from these studies to externally validate these existing prediction models and summarise model performance across studies using random-effects meta-analysis for any, late (after 34 weeks) and early (before 34 weeks) onset pre-eclampsia. If none of the models perform well, we will recalibrate (update), or develop and validate new prediction models using the IPD. We will assess the differential accuracy of the models in various settings and subgroups according to the risk status. We will also validate or develop prediction models based on clinical characteristics only; clinical and biochemical markers; clinical and ultrasound parameters; and clinical, biochemical and ultrasound tests.DiscussionNumerous systematic reviews with aggregate data meta-analysis have evaluated various risk factors separately or in combination for predicting pre-eclampsia, but these are affected by many limitations. Our large-scale collaborative IPD approach encourages consensus towards well developed, and validated prognostic models, rather than a number of competing non-validated ones. The large sample size from our IPD will also allow development and validation of multivariable prediction model for the relatively rare outcome of early onset pre-eclampsia.Trial registrationThe project was registered on Prospero on the 27 November 2015 with ID: CRD42015029349.

Development and validation of multivariable predictive model for thromboembolic events in lymphoma patients.

Lymphoma patients are at increased risk of thromboembolic events but thromboprophylaxis in these patients is largely underused. We sought to develop and validate a simple model, based on individual clinical and laboratory patient characteristics that would designate lymphoma patients at risk for thromboembolic event. The study population included 1,820 lymphoma patients who were treated in the Lymphoma Departments at the Clinics of Hematology, Clinical Center of Serbia and Clinical Center Kragujevac. The model was developed using data from a derivation cohort (n = 1,236), and further assessed in the validation cohort (n = 584). Sixty-five patients (5.3%) in the derivation cohort and 34 (5.8%) patients in the validation cohort developed thromboembolic events. The variables independently associated with risk for thromboembolism were: previous venous and/or arterial events, mediastinal involvement, BMI>30 kg/m(2) , reduced mobility, extranodal localization, development of neutropenia and hemoglobin level < 100g/L. Based on the risk model score, the population was divided into the following risk categories: low (score 0-1), intermediate (score 2-3), and high (score >3). For patients classified at risk (intermediate and high-risk scores), the model produced negative predictive value of 98.5%, positive predictive value of 25.1%, sensitivity of 75.4%, and specificity of 87.5%. A high-risk score had positive predictive value of 65.2%. The diagnostic performance measures retained similar values in the validation cohort. Developed prognostic Thrombosis Lymphoma - ThroLy score is more specific for lymphoma patients than any other available score targeting thrombosis in cancer patients. Am. J. Hematol. 91:1014-1019, 2016. © 2016 Wiley Periodicals, Inc.

External validation of multivariable prediction models: a systematic review of methodological conduct and reporting

BackgroundBefore considering whether to use a multivariable (diagnostic or prognostic) prediction model, it is essential that its performance be evaluated in data that were not used to develop the model (referred to as external validation). We critically appraised the methodological conduct and reporting of external validation studies of multivariable prediction models.MethodsWe conducted a systematic review of articles describing some form of external validation of one or more multivariable prediction models indexed in PubMed core clinical journals published in 2010. Study data were extracted in duplicate on design, sample size, handling of missing data, reference to the original study developing the prediction models and predictive performance measures.Results11,826 articles were identified and 78 were included for full review, which described the evaluation of 120 prediction models. in participant data that were not used to develop the model. Thirty-three articles described both the development of a prediction model and an evaluation of its performance on a separate dataset, and 45 articles described only the evaluation of an existing published prediction model on another dataset. Fifty-seven percent of the prediction models were presented and evaluated as simplified scoring systems. Sixteen percent of articles failed to report the number of outcome events in the validation datasets. Fifty-four percent of studies made no explicit mention of missing data. Sixty-seven percent did not report evaluating model calibration whilst most studies evaluated model discrimination. It was often unclear whether the reported performance measures were for the full regression model or for the simplified models.ConclusionsThe vast majority of studies describing some form of external validation of a multivariable prediction model were poorly reported with key details frequently not presented. The validation studies were characterised by poor design, inappropriate handling and acknowledgement of missing data and one of the most key performance measures of prediction models i.e. calibration often omitted from the publication. It may therefore not be surprising that an overwhelming majority of developed prediction models are not used in practice, when there is a dearth of well-conducted and clearly reported (external validation) studies describing their performance on independent participant data.