Abstract
Risk-prediction models for health outcomes are used in practice as part of clinical decision-making, and it is essential that their performance be externally validated. An important aspect in the design of a validation study is choosing an adequate sample size. In this paper, we investigate the sample size requirements for validation studies with binary outcomes to estimate measures of predictive performance (C-statistic for discrimination and calibration slope and calibration in the large). We aim for sufficient precision in the estimated measures. In addition, we investigate the sample size to achieve sufficient power to detect a difference from a target value. Under normality assumptions on the distribution of the linear predictor, we obtain simple estimators for sample size calculations based on the measures above. Simulation studies show that the estimators perform well for common values of the C-statistic and outcome prevalence when the linear predictor is marginally Normal. Their performance deteriorates only slightly when the normality assumptions are violated. We also propose estimators which do not require normality assumptions but require specification of the marginal distribution of the linear predictor and require the use of numerical integration. These estimators were also seen to perform very well under marginal normality. Our sample size equations require a specified standard error (SE) and the anticipated C-statistic and outcome prevalence. The sample size requirement varies according to the prognostic strength of the model, outcome prevalence, choice of the performance measure and study objective. For example, to achieve an SE < 0.025 for the C-statistic, 60–170 events are required if the true C-statistic and outcome prevalence are between 0.64–0.85 and 0.05–0.3, respectively. For the calibration slope and calibration in the large, achieving SE < 0.15would require 40–280 and 50–100 events, respectively. Our estimators may also be used for survival outcomes when the proportion of censored observations is high.
Highlights
MethodsStatistical Methods in MedicalResearch 30(10)These models are often developed using a regression model that associates the outcome to patient characteristics, the predictor variables
Clinical risk-prediction models are used to predict the risk of either having a health outcome or developing a health outcome in the future using information on patient characteristics
The deterioration in the performance of our formula for the variance of b^CS was expected for very high values of C and was due to the higher efficiency of the linear discriminant analysis (LDA) estimator compared to logistic regression for high values of C: This was confirmed by comparing the efficiency of logistic regression against LDA for a range of values for p and C, when data were generated under data-generating mechanisms (DGMs) 1
Summary
Statistical Methods in MedicalResearch 30(10)These models are often developed using a regression model that associates the outcome to patient characteristics, the predictor variables. The model is fitted to the development data to estimate the regression coefficients which can be used to predict the outcome in new patients. Given the important role of risk models in health care, it is essential to validate risk models, i.e. to assess their predictive performance in either the data used for model development (internal validation) or in a new dataset (external validation). In external validation, the risk model is used to obtain predictions for patients in a new dataset, and the quality of these predictions is assessed using measures of predictive performance, for example, measures of calibration, such as the calibration slope and calibration in the large, and measures of prognostic strength ( called discrimination), such as the C-statistic. The estimated sample sizes for a precision-based calculation, n^req; appðCÞ, n^req; appðbCSÞ and n^req; appðaCLÞ, are obtained using formulae (17), (18) and (19), respectively. The estimated sample size for a power-based calculation, n^req;appðC0; dÞ; is obtained using formula (20)
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
