Clinical Scoring for Risk of Resistant Organisms in Pneumonia: Right Idea, Wrong Interpretation

Abstract

To the Editor—In their recent article, Shorr and colleagues present a new clinical tool to predict risk of drug-resistant bacterial pneumonia in patients presenting to emergency departments [1]. Improving risk assessment and minimizing inappropriate antibiotic use are necessary given the increasing prevalence of resistant pathogens and the risk of selecting for further resistance through overuse of broad-spectrum antibiotics, but this new risk score has several weaknesses. A weighted risk score is an attractive strategy to improve clinical decision making, but the authors overstate its advantages over the existing healthcare-associated pneumonia (HCAP) definition. Although the authors assert that their new risk score is superior to HCAP criteria because it incorporates weighting of risk factors, the cutoff of 0 turns the risk score into a dichotomous tool by which the presence of any risk factor yields a score considered positive rather than a stratified risk score. Under this conclusion, the predictive ability of the new tool is comparable (but not superior) to the current HCAP definition. Using a dichotomous risk tool ignores the information contained in a scoring system, and such a tool cannot be compared directly with the HCAP criteria. To compare the two, cutoff points should be contrasted for persons meeting 1, 2, or more HCAP or risk score criteria. The authors could further improve the predictive ability of their score by presenting likelihood ratios for each of the risk scores of the new tool. The authors present single positive and negative predictive values based on a risk score of 0, but these statistics could be calculated for other scores as well. Because of the effect of prevalence on the performance of diagnostic tests, we are concerned that the prevalence of drug-resistant pathogens reported in the study is inaccurate due to verification bias that occurs when the action of performing a test is affected by the pretest probability of disease [2]. The authors report a 47% prevalence of drug-resistant pathogens, likely an overestimation. A medical history suggesting a person had high risk for drug-resistant bacterial infection might influence physicians’ decisions to obtain cultures, a criterion for inclusion in the study. Persons without significant healthcare exposure might be deemed low-risk by clinicians, who would not order cultures, leading to underrepresentation of these patients in this study. Verification bias results in overestimation of the sensitivity and underestimation of the specificity of the new risk score. Unfortunately it is difficult to confirm that verification bias has occurred; however, the authors could have performed sensitivity analyses or controlled for culture collection to assess this problem. Finally, the most appropriate cutoff for risk score is determined by weighing the costs of further testing or broad-spectrum antibiotic use in patients who do not need them against benefits of appropriate early therapy in those infected with drug-resistant pathogens. This balance will depend on the clinical characteristics and risk factors of the patients, societal views of medical care, and economic considerations. Although the HCAP definition reaches an imperfect balance, the authors’ clinical risk score fails to improve upon that standard.