Comment on Williamson et al. (OpenSAFELY): The Table 2 Fallacy in a Study of COVID-19 Mortality Risk Factors.

Abstract

To the Editor: We write with respect to the recently published work by Williamson et al.1 “OpenSAFELY: factors associated with COVID-19 death in 17 million patients.” We have serious concerns about both the way these results are presented, and how they are likely to be interpreted. Our specific concerns revolve around whether the work is intended by the authors to estimate causal effects, or not—and how, regardless of their intent, it seems likely to us that their work will be interpreted as causal. First, it is notable that the authors describe what they are investigating as risk factors; as others have pointed out,2 the phrase “risk factor” can mean a number of things, including “possible cause under investigation” and “predictor, without attention to cause,” as well as possibly “covariate with a statistically significant association with the outcome.” Here, it is unclear what the authors mean by this phrase. On the causal side of the ledger, the authors describe findings as having “explained” risk, factors being “responsible for [a] reduction” in risk, and of finding “strong evidence of interaction”: all terms which in plain usage imply causal interpretations. Likewise, the authors’ claims that they “have demonstrated…that only a small part of the substantially increased risks of COVID-19 related death among non-white groups…can be attributed to existing disease” and that a “strong association between deprivation and risk was only partly attributable to co-morbidity or other risk factors” are both implicitly causal in use of the word “attribute” and its variants. Similar issues arise with use of the phrase “upstream risk factors”—the word upstream suggesting a temporal ordering, a precondition for causality. Despite these evident implicitly causal (or at least, causal-adjacent) interpretations of their findings, the authors also state that we should not interpret their estimates as causal effects. They note, for example that “the fully adjusted smoking [hazard ratio (HR)] cannot be interpreted causally due to the inclusion of factors likely to mediate smoking effects” and emphasizing “a need for carefully designed causal analyses specifically focusing on the causal effect of smoking on COVID-19 death” and likewise the need for additional analyses exploring other causal relationships. We agree but remain concerned that readers of the work will interpret their results causally. In addition to our concerns noted above about the causal language, readers of this work should be cautioned against committing the Table 2 Fallacy.3 The authors present numerous, mutually adjusted hazard ratios from a single regression model in their Table 2 as well as Figure 3.1 Although such mutually adjusted hazard ratios are not generally interpretable as causal effects, they are often interpreted that way by readers and the press. To interpret the authors’ Table 2, or Figure 3, as if all the mutually adjusted HRs in it have causal interpretations is to commit a classic Table 2 Fallacy. Again, the authors do not make explicit causal interpretations of this work, discussing the figures in Table 2 primarily as (again) “risk factors.” If by risk factors the authors meant this work to be solely or primarily predictive or descriptive in nature then the interpretation and utility of individual HRs (in Table 2) is problematic in a different way. It remains unclear how best to interpret an association adjusted for various factors, or what utility an individual HR from a large model has in predictive terms, when that HR is considered alone. The analyses also fall short of developing a usable clinical prediction model to determine the risk of hospital death in COVID-19 confirmed patients. Although this possibility is underlined by the c-statistic calculated by the authors, such a model would require a more thorough evaluation of predictive performance as well as a clearly defined target population on which the model should be used.4 Indeed, it is unclear how a strictly noncausal risk factor analysis—in which some factors may be causes, and other factors may not—helps us make scientific progress. For example, often risk factor analyses such as this are held up as “hypothesis generating”—typically, implicitly causal hypotheses. “But for the same reasons [THAT] these results cannot be interpreted as causal themselves,” “it is unclear in what way they are useful in forming hypotheses about causal questions.” We are thus concerned that the public and the press may interpret the findings from this study in causal terms; indeed, such interpretations have already been made on Twitter,5 leading to the necessity of correction by an author of the study.6 We urge readers of their work to strictly avoid causal interpretations of their findings. Daniel WestreichDepartment of EpidemiologyUNCChapel Hill, NC[email protected] Jessie K. EdwardsDepartment of EpidemiologyUNCChapel Hill, NC Maarten van SmedenJulius Center for Health Sciences and Primary CareUniversity Medical Center UtrechtUtrecht University, The Netherlands