External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb

Kym I.E Snell,Lucinda Archer,Joie Ensor,Laura J Bonnett,Thomas P.A Debray,Bob Phillips,Gary S Collins,Richard D Riley

doi:10.1016/j.jclinepi.2021.02.011

Abstract

IntroductionSample size “rules-of-thumb” for external validation of clinical prediction models suggest at least 100 events and 100 non-events. Such blanket guidance is imprecise, and not specific to the model or validation setting. We investigate factors affecting precision of model performance estimates upon external validation, and propose a more tailored sample size approach. MethodsSimulation of logistic regression prediction models to investigate factors associated with precision of performance estimates. Then, explanation and illustration of a simulation-based approach to calculate the minimum sample size required to precisely estimate a model's calibration, discrimination and clinical utility. ResultsPrecision is affected by the model's linear predictor (LP) distribution, in addition to number of events and total sample size. Sample sizes of 100 (or even 200) events and non-events can give imprecise estimates, especially for calibration. The simulation-based calculation accounts for the LP distribution and (mis)calibration in the validation sample. Application identifies 2430 required participants (531 events) for external validation of a deep vein thrombosis diagnostic model. ConclusionWhere researchers can anticipate the distribution of the model's LP (eg, based on development sample, or a pilot study), a simulation-based approach for calculating sample size for external validation offers more flexibility and reliability than rules-of-thumb.

Highlights

Sample size “rules-of-thumb” for external validation of clinical prediction models suggest at least 100 events and100 non-events
Factors associated with the precision of model performance estimates: results from simulation study
The standard deviation (σ) of the linear predictor (LP) affected the precision of the C-statistic (Fig. 2, Panels C & D)

Summary

Introduction

Sample size “rules-of-thumb” for external validation of clinical prediction models suggest at least 100 events and100 non-events. An important part of prediction model research is assessing the predictive performance of a model, in terms of whether the model’s predicted risks: (i) discriminate between individuals that have the outcome and those that do not, and (ii) calibrate closely with observed risks (ie, predicted risks are accurate). Examining clinical utility (eg, a model’s net benefit) is important if the model is to be Consider a prediction model, developed using logistic regression for a binary outcome, that is to be externally validated. It will take the form, log pi 1 − pi. The predictive performance of a model is usually evaluated by estimating measures of calibration, discrimination and clinical utility, as defined in Box 1

Results

Discussion

Conclusion