In Reply.

Abstract

The authors thank Dr. Byrne and colleagues for an insightful response to our recent meta-analysis on the prognostic accuracy of the HEART score for prediction of major adverse cardiac events (MACE) in patients presenting with chest pain.1 The authors make excellent points related to the evaluation of prognostic scores, particularly as it relates to clinical decision making. We feel that their editorial does highlight important complementary discussion points to our article. First, the authors correctly point out that acute coronary syndrome (ACS) remains a largely clinical diagnosis. Most importantly, and contrary to the authors’ assertion, we did not suggest in our article that the HEART score should be used to rule out the diagnosis of ACS, and in fact, the tool was not designed for this purpose.2 It is widely agreed that patients with a diagnosis of ACS should be admitted to hospital for monitoring, further testing, and intervention. In patients for whom ACS has been ruled out, the HEART score seeks to answer the question of whether that patient should receive further downstream testing in the short term (and, depending on the available resources, hospital admission) or whether they can be safely discharged for follow-up at a later date. Given concerns regarding medicolegal liability, at present many of these patients do undergo further testing, often with significant resource utilization and potential adverse events from the testing itself.3, 4 The authors correctly identify that there are significant issues with using the MACE composite outcome, namely, as it relates to incorporation and verifications biases, which are discussed in our study.5 We still felt that this was the best outcome to utilize, given its ubiquity in the emergency medicine and cardiology literature, as well as the fact that emergency medicine clinical decision instruments often strive to evaluate multiple clinically important outcomes,6 and this approach was also taken in another similar meta-analysis of the prognostic accuracy of the HEART score.7 We believe that this is because emergency department (ED) physicians strive to not only predict mortality, but also other factors that would influence disposition. The authors additionally identify important biases that may occur in the prediction of prognosis by a clinical decision instrument. They note the important nature of spectrum bias. That is, if the prevalence of the target condition varies significantly between study populations, it can influence the accuracy of the parameters being investigated. This issue also arises in diagnostic test accuracy meta-analyses,8 because as the authors correctly point out, sensitivity and specificity can vary in populations with markedly different prevalence of the target condition. To that end, the authors state that our study should have preferentially presented predictive values, as opposed to sensitivity and specificity. We do agree that predictive values would be beneficial in providing greater understanding of how the HEART score performs. In Tables S7 and S8, we did present the predictive values in the form of posttest probabilities. The concept of predictive values pertains to the prevalence within a given study population but may not be properly defined across a collection of studies with varying prevalence. Therefore, predictive values cannot be meta-analyzed directly. In the context of our study, we considered it inappropriate to fix prevalence at a specific value. Instead, we could assume a series of prevalence/pretest probability values, while having sensitivity and specificity fixed at the summary estimates from meta-analyses. The posttest probabilities of MACE, given the summary estimates of sensitivity and specificity of HEART ≥ 4 (or ≥ 7) and the pretest probabilities (or prevalence) could be considered as the positive predictive values. The posttest probabilities of MACE given the summary estimates of sensitivity and specificity of HEART < 4 (or < 7) and the pretest probabilities could be considered as the one minus negative predictive values. Additionally, sensitivity and specificity were the most commonly reported metrics in our included studies, which likely speaks to the way that the test is utilized in practice. In ED situations where a missed outcome (false negative) is particularly important, such as MACE, high sensitivity is thought to be the most valued metric in clinical decision instruments.6 All of our studies were conducted in the ED, and the large majority evaluated undifferentiated patients with chest pain. While the authors highlight a single study where the prevalence of MACE was particularly low,9 the majority of our included studies (73.3%) had a prevalence of MACE between 10 and 20%, in keeping with the most commonly cited short-term rates for MACE among ED patients without alternative diagnoses for their chest pain.10 Only four studies (13.3%) had a prevalence of MACE > 20%. While changes in prevalence can produce changes in sensitivity and specificity, Leeflang et al.11 found that this occurred in only 35% of the meta-analyses they investigated, and the median range in prevalence of their included meta-analyses was 37%. Furthermore, Leeflang et al. noted that changes in prevalence influence patients without the outcome of interest more than those with it, suggesting that it influences specificity more than sensitivity.11 We agree with the authors that the sensitivity of our pooled estimate does differ from the sensitivity identified in their single cited example of a low-risk population, but the magnitude of the true difference is unknown, and therefore evaluation of the HEART score in low- and high-risk populations should be an area of ongoing research. Finally, the authors comment upon the utilization of the HEART score as a “test.” We would argue that the HEART score was created to function as a clinical decision instrument. While the HEART score was not created through the well-accepted methods for clinical decision instruments (contrary to the authors’ assertion, there was never a true derivation population used for creation of the tool),6 its utilization in clinical practice reflects the nature of a decision instrument.2 This is why, also contrary to the authors’ assertion, the large majority of the studies we evaluated in our review do present metrics for sensitivity, specificity, and likelihood ratios. To that end, our study simply attempted to meta-analyze and present the pooled metrics from these studies, particularly given the recommendation from the American Heart Association and the American College of Cardiology that clinical decision instruments should be used in the evaluation of ED chest pain patients.12 While the HEART score did outperform the Thrombolysis in Myocardial Infarction (TIMI) score in our meta-analysis and should be the preferred decision instrument used in the evaluation of chest pain patients, there are important concerns. First, the pooled sensitivity of the tool (~96%) is not sufficiently high for use in isolation.6, 13 Second, there are emerging concerns related to inter-rater reliability in ED scoring of the HEART score, likely owing to the subjective nature of some components.14 This can therefore influence estimates of HEART score accuracy. Finally, the resources available to individual providers may vary. At some centers, patients may wait months for outpatient testing after ED assessment for chest pain. In such instances, the HEART score's ability to predict short-term MACE is less useful. Taken together, we do agree with the authors that the HEART score should not be used to “rule out” future MACE but should rather be utilized in conjunction with clinical gestalt and evaluated in the context of available resources. There remain significant concerns with the practical application of the HEART score in isolation.