Abstract
ObjectivesOur aim was to improve meta-analysis methods for summarizing a prediction model's performance when individual participant data are available from multiple studies for external validation. Study Design and SettingWe suggest multivariate meta-analysis for jointly synthesizing calibration and discrimination performance, while accounting for their correlation. The approach estimates a prediction model's average performance, the heterogeneity in performance across populations, and the probability of “good” performance in new populations. This allows different implementation strategies (e.g., recalibration) to be compared. Application is made to a diagnostic model for deep vein thrombosis (DVT) and a prognostic model for breast cancer mortality. ResultsIn both examples, multivariate meta-analysis reveals that calibration performance is excellent on average but highly heterogeneous across populations unless the model's intercept (baseline hazard) is recalibrated. For the cancer model, the probability of “good” performance (defined by C statistic ≥0.7 and calibration slope between 0.9 and 1.1) in a new population was 0.67 with recalibration but 0.22 without recalibration. For the DVT model, even with recalibration, there was only a 0.03 probability of “good” performance. ConclusionMultivariate meta-analysis can be used to externally validate a prediction model's calibration and discrimination performance across multiple populations and to evaluate different implementation strategies.
Highlights
A crucial part of medical research is to develop risk prediction models
We propose multivariate meta-analysis for jointly synthesizing discrimination and calibration performance, while accounting for their correlation
We show that the multivariate approach summarizes a prediction model’s average discrimination and calibration performance and quantifies the heterogeneity in performance across populations
Summary
Our aim was to improve meta-analysis methods for summarizing a prediction model’s performance when individual participant data are available from multiple studies for external validation
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have