An exemplary reanalysis of coronary computed tomography angiography diagnostic meta-analyses shows insufficient data sharing and incorrect sensitivity and specificity estimates

Abstract

ObjectivesTo systematically evaluate the reproducibility of primary data and, the reproducibility and correctness of pooled sensitivity and specificity estimates reported in a sample of diagnostic meta-analyses. Study Design and SettingWe conducted an exemplary systematic review of diagnostic meta-analyses comparing coronary computed tomography angiography to invasive coronary angiography in patients with suspected coronary artery disease. The objectives were to assess 1) the reproducibility of contingency tables, 2) the reproducibility of pooled sensitivity and specificity, and 3) differences to reported results when applying a recommended bivariate binomial model for pooling sensitivity and specificity. Therefore, we reproduced the contingency tables and recalculated sensitivity and specificity by utilizing both the pooling method of each meta-analysis and a bivariate binomial model. We used linear trends to assess the improvement of these objectives over time. ResultsWe identified 38 diagnostic meta-analyses, each including on average 19 primary studies (range: 3 to 89 studies; total: 715—including duplicates) with an average of approximately 1800 patients per meta-analysis (range: 118 to 7516 patients). For 31 meta-analyses (82%, 95% CI: 65%, 91%), the contingency tables were reproducible; however, only 15 published them. Using the pooling method of each meta-analysis, we obtained comparable recalculated sensitivities/specificities for 28 meta-analyses (74% [57%, 86%]). Only 11 meta-analyses pooled sensitivity/specificity using a bivariate binomial model (29% [16%, 46%]). When all meta-analyses were pooled with this model, published sensitivities/specificities were confirmed for 19 of 38 meta-analyses (50% [34%, 66%]). There was only marginal improvement in data availability and application of recommended pooling methods over time. ConclusionData sharing should become standard practice along with the use of appropriate pooling methods. Journal publication requirements may play a key role in enhancing the quality of scientific reporting and methodological standards which may lead to more reliable and consistent outcomes.The ability to reproduce sensitivity and specificity estimates in diagnostic imaging meta-analyses is dependent on the availability of contingency tables and the explicit reporting of pooling methods and software used.