Abstract 195: External Validations of Clinical Prediction Models for Cardiovascular Disease: A Field Synopsis

Abstract

Background: Clinical prediction models (CPMs) hold the potential to improve decision-making and individualize care for patients with cardiovascular disease (CVD); however, it is increasingly recognized that CPM performance may decrease when validated externally. Here we systematically describe published external validations of CPMs for patients with CVD, with specific attention to changes in CPM performance. Methods: A systematic review was conducted using a citation search to identify external validations of CPMs in the Tufts Predictive Analytics and Comparative Effectiveness (PACE) CPM Registry, a comprehensive database of CPMs for patients with CVD published between 1990 and May 2015. Information on CPM performance in the original derivation database as well as during subsequent external validations (through March 2017) was extracted. The percent change in discrimination was calculated as [(Validation AUC - 0.5) - (Derivation AUC - 0.5) / (Derivation AUC - 0.5) * 100]. Results: The Registry includes 1083 CPMs for CVD. 1555 external validations were identified. On average there were 1.4 validations/ de novo CPM (range 0-83). 758 (70%) of the CPMs have never been externally validated. The median external validation sample size was 1215 (IQR 430, 6509) and the median number of events 59 (21, 209). 88% (1368) of the external validations report area under the receiver operating characteristic curve (AUROC). 59% (917) report some measure of CPM calibration. The median external validation AUROC was 0.74 (0.69-0.81) and the median percent change in discrimination was -14.7% (-34.1%, +0.8%). 29.3% (455) of model validations showed CPM discrimination at or above the performance reported in the derivation dataset. Conclusion: While numerous CPMs exist for patients with CVD, most have never been externally validated. CPMs generally show substantially worse discrimination in external validations compared to the derivation datasets.