Abstract
It is widely acknowledged that the predictive performance of clinical prediction models should be studied in patients that were not part of the data in which the model was derived. Out‐of‐sample performance can be hampered when predictors are measured differently at derivation and external validation. This may occur, for instance, when predictors are measured using different measurement protocols or when tests are produced by different manufacturers. Although such heterogeneity in predictor measurement between derivation and validation data is common, the impact on the out‐of‐sample performance is not well studied. Using analytical and simulation approaches, we examined out‐of‐sample performance of prediction models under various scenarios of heterogeneous predictor measurement. These scenarios were defined and clarified using an established taxonomy of measurement error models. The results of our simulations indicate that predictor measurement heterogeneity can induce miscalibration of prediction and affects discrimination and overall predictive accuracy, to extents that the prediction model may no longer be considered clinically useful. The measurement error taxonomy was found to be helpful in identifying and predicting effects of heterogeneous predictor measurements between settings of prediction model derivation and validation. Our work indicates that homogeneity of measurement strategies across settings is of paramount importance in prediction research.
Highlights
Prediction models have an important role in contemporary medicine by providing probabilistic predictions of diagnosis or prognosis.[1]
Heterogeneity of predictor measurements across settings can have a substantial impact on the out-of-sample performance of a prediction model
When predictor measurements are more precise at derivation compared to validation, model discrimination and accuracy at validation deteriorate, and the provided predicted probabilities are too extreme, similar to when a model is overfitted with respect to the derivation data
Summary
Prediction models have an important role in contemporary medicine by providing probabilistic predictions of diagnosis or prognosis.[1]. While out-of-sample predictive performance is in general expected to be lower than performance estimated at derivation,[1] large discrepancies are often contributed to suboptimal modeling strategies in the derivation of the model[3,4,5] and differences between patient characteristics in derivation and validation samples.[6,7] Another potential source of limited out-of-sample performance is when predictors are measured differently at derivation than at (external) validation. We investigate the out-of-sample performance of a clinical prediction model in situations where predictor measurement strategies at the model derivation stage differed from measurement strategies at the model validation stage.
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have