Picking a winner among decision aids

Abstract

We are concerned about what we believe are a series of erroneous comparisons made by Bandiera et al1 in their article in the September 2003 issue of Annals (article #293) regarding the Canadian C-Spine rule and the National Emergency X-Radiography Utilization Study (NEXUS) decision instrument.2 We also question including these in the Discussion section of a research article that did not deal directly with these issues. Because of space limitations in a letter to the editor, we are unable to address completely our concerns, but we would like to make the following observations:1.Comparing the reported specificities of these 2 instruments is much like comparing apples to oranges, because each measured different things. The Canadian study included all patients who sustained neck trauma, including a sizeable proportion who never had radiographs taken, whereas the NEXUS investigators specifically excluded such patients. There are many lines of evidence to suggest that, had the 2 studies measured specificity in the same manner, the results would have been extremely similar. Readers should note that the effect on current ordering expected with application of each of the instruments is very similar, even though current ordering patterns in NEXUS were more efficient than in the Canadian study (achieving almost twice as high a fracture rate) and despite the fact that film ordering in NEXUS had already decreased substantially after publication of our derivation set 8 years earlier.32.We believe the authors also make an unequal comparison in their discussion of sensitivity, referring to sensitivity for identifying “significant” injury when discussing the Canadian rule, but not when discussing NEXUS. Given that no instrument can be expected to pick up every single case of a disease, it is not surprising that NEXUS, with more than 4 times as many patients with significant injury, was the first of the 2 to identify “false negative” cases (although both of the 2 NEXUS “misses” clearly represented misclassification rather than an actual failure of the NEXUS instrument). The 95% confidence intervals for sensitivity for clinically significant injury are 98% to 100% for the Canadian rule (in the derivation set only) and 98.6% to 100% for the NEXUS decision instrument.3.The authors cite work that has not been published in the peer-reviewed literature, asserting that “a retrospective application of the NEXUS criteria to a CCC study population revealed a sensitivity of only 93%.” This is statistically incompatible with the results of NEXUS and is explained by the fact that they failed to evaluate the true NEXUS instrument. Indeed, the Canadian authors themselves expressed concerns when Annals published 2 articles claiming to show that the Ottawa Ankle Rules and Knee Rules were less than perfect.4,5 In both instances, they appropriately noted that the studies in question had not tested the “real” Ottawa rules, pointing out that they used imprecise variations of the respective rules and flawed retrospective methodologies.2 However, these same shortcomings—a review of data forms designed for an entirely different purpose, containing a large number of other data points, and substituting surrogates for at least 3 of the 5 precise NEXUS variables—are also true of the authors' current attempt to validate the NEXUS decision instrument.4.The authors then explain this by citing the “low interobserver reliability” of the NEXUS criteria, again based on their own work, while ignoring the high levels of agreement we repeatedly found in our own assessments of the same criteria.6,7 The differences in measured interobserver reliability lend further credence to the notion that the Canadians measured something quite different than the actual criteria as defined in NEXUS. More importantly, the only reason to measure interobserver variability is as a surrogate, a marker intended to try to predict how a given clinical finding might perform when applied by a large number of observers. In actual fact, NEXUS was applied by hundreds of emergency physicians to more than 34,000 patients, and its very high sensitivity was achieved by all those different physicians applying the criteria as they saw fit.5.We disagree, therefore, with the authors' conclusion that “wide application of the NEXUS rule should be preceded by further investigation … in a broad population.” As noted, this has already been done as part of the NEXUS project, deliberately designed to include an enormous range of physicians working in a wide range of hospitals. The same cannot be said for the Canadian study, which was derived in what appears to be a fairly homogeneous group of hospitals.