Minimum sample size for external validation of a clinical prediction model with a continuous outcome.

Lucinda Archer,Mohammed T Hudda,Richard D Riley,Kym I E Snell,Joie Ensor,Gary S Collins

doi:10.1002/sim.8766

Lucinda Archer, Mohammed T Hudda + Show 4 more

Open Access

https://doi.org/10.1002/sim.8766

Copy DOI

Abstract

Clinical prediction models provide individualized outcome predictions to inform patient counseling and clinical decision making. External validation is the process of examining a prediction model's performance in data independent to that used for model development. Current external validation studies often suffer from small sample sizes, and subsequently imprecise estimates of a model's predictive performance. To address this, we propose how to determine the minimum sample size needed for external validation of a clinical prediction model with a continuous outcome. Four criteria are proposed, that target precise estimates of (i) R2 (the proportion of variance explained), (ii) calibration-in-the-large (agreement between predicted and observed outcome values on average), (iii) calibration slope (agreement between predicted and observed values across the range of predicted values), and (iv) the variance of observed outcome values. Closed-form sample size solutions are derived for each criterion, which require the user to specify anticipated values of the model's performance (in particular R2 ) and the outcome variance in the external validation dataset. A sensible starting point is to base values on those for the model development study, as obtained from the publication or study authors. The largest sample size required to meet all four criteria is the recommended minimum sample size needed in the external validation dataset. The calculations can also be applied to estimate expected precision when an existing dataset with a fixed sample size is available, to help gauge if it is adequate. We illustrate the proposed methods on a case-study predicting fat-free mass in children.

Highlights

Clinical prediction models provide individualized outcome predictions to inform patient counseling and clinical decision making, such as treatment and monitoring strategies.[1,2,3] Depending on the context, they may be referred to as clinical prediction tools, diagnostic or prognostic models, risk scores, and prognostic indices, among other names
We propose criteria to determine the minimum sample size needed for external validation of a clinical prediction model with a continuous outcome
The approach requires a separate sample size calculation for each criterion, and the largest sample size calculated provides the minimum needed for the external validation study

Summary

Introduction

Clinical prediction models provide individualized outcome predictions to inform patient counseling and clinical decision making, such as treatment and monitoring strategies.[1,2,3] Depending on the context, they may be referred to as clinical prediction tools, diagnostic or prognostic models, risk scores, and prognostic indices, among other names. They are typically developed using a regression framework, which provides an equation to predict the outcome conditional on the values of multiple predictors (variables, covariates). The outcome may relate to something current (eg, fat mass level at present) or in the future (eg, pain score at 1 month after a back injury)

Objectives

Findings

Discussion

Conclusion