The failure rate does not equal the false‐negative rate: A call for tailoring diagnostic strategy validation in low prevalence populations

Abstract

Over the past decade, various new diagnostic strategies have been tested and validated for the diagnosis of pulmonary embolism (PE) in the emergency department or in primary care. The main goal of a new strategy is to safely decrease the need for further investigation, particularly imaging studies (usually computed tomographic pulmonary angiogram), and reducing overall resource consumption including time spent in the emergency department.1.Newman D.H. Schriger D.L. Rethinking testing for pulmonary embolism: less is more.Ann Emerg Med. 2011; 57: 622-627.e3https://doi.org/10.1016/j.annemergmed.2011.04.014Abstract Full Text Full Text PDF PubMed Scopus (34) Google Scholar The success of these recent strategies has resulted in excellent sensitivities with subsequent very high negative predictive values and low false‐negative rates. Consequently, the further development of any new diagnostic strategies should not focus on improving the sensitivity or the overall discrimination, but rather on improving specificity without impairing sensitivity. To validate the safety of a strategy, a maximum acceptable failure rate is regularly redefined. From a former threshold between 2.7% and 4% based on pulmonary angiogram's performances, new recommendations suggested that the maximum threshold should be dependent on the prevalence of PE in the tested population.2.van Strijen M.J.L. de Monyé W. Schiereck J. et al.Single‐detector helical computed tomography as the primary diagnostic test in suspected pulmonary embolism: a multicenter clinical management study of 510 patients.Ann Intern Med. 2003; 138: 307-314https://doi.org/10.7326/0003‐4819‐138‐4‐200302180‐00009Crossref PubMed Scopus (231) Google Scholar In 2017, the SSC of the ISTH recommended that the maximal acceptable failure rate should be 1.82% + 0.0053% × prevalence.3.Dronkers C.E.A. van der Hulle T. Le Gal G. et al.Towards a tailored diagnostic standard for future diagnostic studies in pulmonary embolism: communication from the SSC of the ISTH.J Thromb Haemost JTH. 2017; 15: 1040-1043https://doi.org/10.1111/jth.13654Crossref PubMed Scopus (65) Google Scholar Therefore, in a low prevalence population, a new diagnostic strategy to rule out PE will be validated if the upper bound of the 95% confidence interval (CI) of the failure rate is below 1.82. It is critical, however, to consider what we define as the “failure rate.” The current definition is the number of missed PE (numerator) divided by the total number of patients in whom the strategy has been evaluated (denominator). This highlights a serious shortcoming: it is totally dependent on the tested population, which was addressed in the 2017 SSC recommendations. Another serious shortcoming is that this definition omits an important variable: the number of patients in whom the strategy has actually changed the workup strategy (which can be partially capture by the net reclassification index). For example, imagine testing a strategy that will adjust the D‐dimer threshold in a population of 1000 subjects, but in which only 100 patients fit the criteria in which the threshold should be changed. Therefore, the recommended denominator for failure rate calculation is 1000, despite the fact that it is clinically relevant to only 100 patients. Consequently, we argue that, although the denominator can include all patients processed by the algorithm, its evaluation in the subgroup of patients affected by the strategy, if taken alone, will have a higher failure rate. In this example, if the tested strategy missed 10 PEs, the conventional failure rate would be 10/1000 = 1%. This would will be considered safe, because the upper bound of the 95% CI is at 1.83. However, if the 10 missed PEs are counted in the subpopulation in which D‐dimer threshold should have been adjusted (i.e., the population in which the strategy actually had an impact), the rate of missed PE would be 10% (95% CI, 10–34). It is therefore critically important to report the failure rate in the subpopulation in which the strategy was effectively applied. For example, trials that assessed the safety of the pulmonary embolism rule‐out criteria (PERC) strategy should be evaluated on the failure rate among patients with a PERC of zero. Another example (among many others) is the Van der Pol et al. study that assessed the safety of the YEARS algorithm in pregnant women, and which reported a failure rate of 0.21% (95% CI, 0.04–1.2). However, the failure rate among women who actually had a change of strategy (no YEARS criteria and D‐dimer <1000 ng/ml) was 1/164 (i.e., with an upper bound of the 95% CI at 3). Therefore, this questions the safety of this strategy. We could imagine a “heads or tails” strategy to rule out PE in a low‐prevalence population. If we were to flip a coin in a population of 4000 patients with a 2.5% rate of PE (which is consistent with the prevalence found in several studies), this would conclude (approximately) that PE can be ruled out in 2000 random patients.4.Pernod G. Caterino J. Maignan M. et al.D‐Dimer use and pulmonary embolism diagnosis in emergency units: why is there such a difference in pulmonary embolism prevalence between the United States of America and countries outside USA?.PLoS One. 2017; 12https://doi.org/10.1371/journal.pone.0169268Crossref PubMed Scopus (29) Google Scholar, 5.Freund Y. Cachanado M. Aubry A. et al.Effect of the pulmonary embolism rule‐out criteria on subsequent thromboembolic events among low‐risk emergency department patients: the PROPER randomized clinical trial.JAMA. 2018; 319: 559-566https://doi.org/10.1001/jama.2017.21904Crossref PubMed Scopus (84) Google Scholar The prevalence would remain 2.5% in both groups. However, if the denominator were incorrectly assumed to be 4000, the failure rate would be assumed to be 1.25% (with an upper bound of the 95% CI below 1.8). With this logic error in place, the SSC recommendations could be inappropriately used to assert flipping a coin is a safe strategy. One should question whether we are ready to adopt a strategy that will miss 50% of PEs where the diagnostic work up is changed. Recently derived strategies (e.g., PERC, YEARS, pulmonary embolism graduated d‐dimer) targeted a population with low prevalence of PE to reduce the need for computed tomographic pulmonary angiogram.5.Freund Y. Cachanado M. Aubry A. et al.Effect of the pulmonary embolism rule‐out criteria on subsequent thromboembolic events among low‐risk emergency department patients: the PROPER randomized clinical trial.JAMA. 2018; 319: 559-566https://doi.org/10.1001/jama.2017.21904Crossref PubMed Scopus (84) Google Scholar, 6.Kline J.A. Courtney D.M. Kabrhel C. et al.Prospective multicenter evaluation of the pulmonary embolism rule‐out criteria.J Thromb Haemost JTH. 2008; 6: 772-780https://doi.org/10.1111/j.1538‐7836.2008.02944.xCrossref PubMed Scopus (283) Google Scholar, 7.van der Pol L.M. Tromeur C. Bistervels I.M. et al.Pregnancy‐adapted YEARS algorithm for diagnosis of suspected pulmonary embolism.N Engl J Med. 2019; 380: 1139-1149https://doi.org/10.1056/NEJMoa1813865Crossref PubMed Scopus (0) Google Scholar, 8.Kearon C. de Wit K. Parpia S. et al.Diagnosis of pulmonary embolism with d‐Dimer adjusted to clinical probability.N Engl J Med. 2019; 381: 2125-2134https://doi.org/10.1056/NEJMoa1909159Crossref PubMed Scopus (151) Google Scholar It is unclear whether the interpretation and analysis of these studies should include patients with non‐low prevalence (i.e., patients in whom the strategy will not change the workup strategy). For a better, more transparent, and comprehensive evaluation of a tested diagnostic strategy, we believe that results should always include the 2 × 2 diagnostic contingency matrix, and the safety should focus on one major indicator: number of missed PEs divided by number of patients where the strategy was actually applied (i.e., failure rate in the population of interest). All authors declare they have no conflict of interest with this manuscript. All authors participated in the writing of this manuscript, and approved the final version. YF drafted the paper. All co‐authors provided substantial revisions. All authors approved the final version of the manuscript.