Accounting for treatment use when validating a prognostic model: a simulation study

Romin Pajouheshnia,Karel G M Moons,Johannes B Reitsma,Linda M Peelen,Rolf H H Groenwold

doi:10.1186/s12874-017-0375-8

Romin Pajouheshnia, Karel G M Moons + Show 3 more

Open Access

https://doi.org/10.1186/s12874-017-0375-8

Copy DOI

Abstract

BackgroundPrognostic models often show poor performance when applied to independent validation data sets. We illustrate how treatment use in a validation set can affect measures of model performance and present the uses and limitations of available analytical methods to account for this using simulated data.MethodsWe outline how the use of risk-lowering treatments in a validation set can lead to an apparent overestimation of risk by a prognostic model that was developed in a treatment-naïve cohort to make predictions of risk without treatment. Potential methods to correct for the effects of treatment use when testing or validating a prognostic model are discussed from a theoretical perspective.. Subsequently, we assess, in simulated data sets, the impact of excluding treated individuals and the use of inverse probability weighting (IPW) on the estimated model discrimination (c-index) and calibration (observed:expected ratio and calibration plots) in scenarios with different patterns and effects of treatment use.ResultsIgnoring the use of effective treatments in a validation data set leads to poorer model discrimination and calibration than would be observed in the untreated target population for the model. Excluding treated individuals provided correct estimates of model performance only when treatment was randomly allocated, although this reduced the precision of the estimates. IPW followed by exclusion of the treated individuals provided correct estimates of model performance in data sets where treatment use was either random or moderately associated with an individual's risk when the assumptions of IPW were met, but yielded incorrect estimates in the presence of non-positivity or an unobserved confounder.ConclusionsWhen validating a prognostic model developed to make predictions of risk without treatment, treatment use in the validation set can bias estimates of the performance of the model in future targeted individuals, and should not be ignored. When treatment use is random, treated individuals can be excluded from the analysis. When treatment use is non-random, IPW followed by the exclusion of treated individuals is recommended, however, this method is sensitive to violations of its assumptions.

Highlights

Prognostic models often show poor performance when applied to independent validation data sets
We provide a detailed explanation of when and how treatment use in a validation set can bias the estimation of the performance of a prognostic model in future targeted individuals and compare different analytical approaches to correctly estimate the performance of a model using a partly treated validation data set in a simulation study
A prognostic model developed for making predictions of risk without that treatment will erroneously appear to overestimate risk in a partially treated validation set, regardless of how treatments have been allocated

Summary

Introduction

Prognostic models often show poor performance when applied to independent validation data sets. When additional treatment use in a validation set (compared to the development set) results in a markedly lower incidence of the outcome under prediction, the predictive performance of the model will likely be affected. A challenge arises when a prognostic model has originally been developed in order to make predictions of “untreated risks”, i.e. predictions of an individual’s prognosis without certain treatments, to guide the decision to initiate those treatments in future targeted individuals. These models should be validated in data sets in which individuals remain untreated with those specific treatments throughout follow-up, socalled treatment-naïve populations. The use of such treatment-naïve populations is uncommon and poor performance of a prognostic model seen in a validation study could be directly attributed to treatment use in the validation data set [11, 12]

Methods

Results

Conclusion