Predictive Performance of Three Multivariate Difficult Tracheal Intubation Models: A Double-Blind, Case-Control Study

Abstract

To the Editor: Naguib et al. (1) evaluated the performance of three multivariate models to predict the absolute probability of a difficult intubation. They subsequently derived a new model. We acknowledge the relevance and potential utility of a new prediction model for difficult intubation with higher performance, also in view of a recent meta-analysis on this topic (2). However, we believe that Naguib et al.’s study has two important design errors that should make practitioners cautious about drawing inferences. The authors used a 1:1 matched case–control design, an improper design for the study question. First, because of this design, the blinded investigator who assessed the presence or absence of intubation difficulty knew beforehand that one of each two patients (50%) would be difficult to intubate. In daily practice, however, the average incidence of difficulty is only about 6% (2,3). It was therefore, even more curious when the authors reported an incidence of only 0.13% (97/73,696). This low incidence was probably caused by under-reporting of cases (i.e., cases of difficult intubation were not reported to the authors during the study). Second, a case–control design is not a proper design for answering diagnostic and prognostic questions or for developing and validating prediction models (4,5). In a case– control design, investigators are free to choose the number of cases and controls. Therefore, they can “manipulate” the a priori probability of the outcome and thus also the posterior probabilities, i.e., the positive and negative predictive values (PPV and NPV). Only data from cohort studies allow derivation of absolute outcome probabilities. When the original cohort size from which the cases and controls were sampled is known, one can use a weighting method to derive a prediction model that allows for estimating absolute probabilities (a nested case–control design) (4). Although the authors reported the original cohort size and number of patients with a difficult intubation (97 in 73,696 patients), they seemed to use only the data from the 194 cases and controls, without weighting the cases and controls. Thus, the intercept of their model and the predictive values presented in Table 4 of their article are biased. This bias can be illustrated with data from their own study, using the predictor Mallampati score (MP1). The authors selected 97 patients with a difficult intubation (cases) and 97 without a difficult intubation (controls) from a cohort. The ratio among controls was 51 (53%) MP1 vs 46 (47%) >MP1, and among cases was 7 (7%) MP1 vs 90 (93%) >MP1 (see Table 1, numbers without parentheses). The pretest probability of the outcome was 50% (97/194), the probability for patients classified as MP1 (NPV) was 7/58 = 12% and for patients classified as >MP1 (PPV) was 90/136 = 66%. If the investigators had not used a 1:1 ratio, but had instead used a 1:2 ratio with 194 controls (Table 1, numbers within parenthesis), the a priori probability would become 33% (97/291), the NPV would decrease to 6% (7/109), and the PPV to 49% (90/182). In fact, each different ratio of cases and controls will give different overall Mallampati, class-specific, outcome probabilities. This applies not only to dichotomous, but also to continuous predictors (like Interincisor gap), as well as to combinations of predictors by using a multivariable model.Table 1: Association of Mallampati Class 1 with Difficult Intubation for Two Different Ratios of Cases and ControlsIn conclusion, the case–control design used by the authors has resulted in unreliable estimates of the probability of a difficult intubation, both before and after testing. However, a new and correct analysis can be performed easily after the cases and controls are weighted for their sampling fraction, given that the reported incidence of 0.13% is true and not caused by under-reporting of cases (6). Wilton A. van Klei Department of Anesthesia The Ottawa General Hospital Ottawa, Canada Department of Perioperative Care and Emergency Medicine [email protected] Cornelis J. Kalkman Department of Perioperative Care and Emergency Medicine Karel G. M. Moons Department of Perioperative Care and Emergency Medicine Julius Centre for Health Sciences and Primary Care University Medical Center Utrecht, The Netherlands