Commentary: Observational studies and the art of accurately measuring influenza vaccine benefits

Abstract

As Orenstein and colleagues point out in an elegant paper in this issue, observational studies that compare the incidence of influenza-like illness in vaccinated and unvaccinated groups are theoretically prone to underestimating the vaccine’s true effectiveness (VE). This is because many other respiratory pathogens cause similar symptoms; these other infections form a sizeable background of influenza-like illness cases in both case and control groups that are not preventable by influenza vaccination. Orenstein et al. reasonably call for laboratory confirmed endpoints, which have rarely been obtained in observational studies of influenza vaccine effectiveness in populations targeted for vaccination. These theoretical effects of low endpoint specificity are perhaps not so novel, but this paper and its careful quantification of the issue is timely because the problems of interpreting results from studies that use low-specificity end-points seem to have been all but forgotten in the contemporary literature. In a perfect world, there would be plenty of ‘Gold standard’ evidence from randomized placebo-controlled clinical trials (RCTs) that measure vaccine efficacy using highly specific laboratory-confirmed influenza endpoints. But these are scarce in the influenza literature. A placebo control group is simply not an option when studying vaccine benefits in populations that are already recommended for vaccine (such as seniors and persons with high-risk conditions). For that reason, observational studies have long made up the largest part of the evidence base, especially for influenza vaccine benefits in seniors. Assuming a near-perfect world, Orenstein et al. theoretically explore the expected performance of cohort and case-control study designs that use laboratory-confirmed endpoints, the latter with two different approaches to control selection. They focused on the consequences of less-than-perfect sensitivity and specificity of the rapid laboratory tests (an increasingly popular choice over culture-confirmation as the price of these kits falls) and on the prevalence of influenza relative to other respiratory pathogens. However, their simulations did not explore the possibility of selection bias leading to various degrees of mismeasurement. Orenstein et al. first explored a base-case scenario of a paediatric population, assuming realistic parameters of attack rates of influenza (15%) and other respiratory pathogens (30%), and rapid tests with 80% sensitivity and 90% specificity. In this scenario, all observational study designs performed about the same, although case-control studies using non-lab-positive controls had a slight advantage. All underestimate the true VE by about 15%; for example, a true VE of 70% would be measured at 56–60%. But as can be seen from results of the sensitivity analyses (Figure 2 in ref.), under a scenario in which the influenza attack rate is low relative to other respiratory illness (5% vs 30%), the VE would be substantially underestimated; for example, a VE of 70% would be measured at 40% or lower. So the good news from Orenstein et al. is that observational studies can generate reasonably accurate VE estimates, as long as laboratory tests are highly (590%) specific and as long as the incidence of influenza is relatively high. But as the authors point out, the bad news is that the use of endpoints with low specificity—such as clinical influenza-like illness without laboratory confirmation—is not appropriate. The consequences can be seen by perusing their Figure 2. If the specificity of the endpoint is 70%, for example, the underestimation is profound (measured VE at 10%). With extrapolation of Orenstein’s findings to a lower level of sensitivity of the outcome, it would naturally produce far worse results. Halloran and Longini previously developed a classy framework for cohort studies to analytically adjust measured VE for less specific outcomes in order to provide true VE estimates; however, these authors also did not consider the possible complications of selection bias. In the end, however, we must come back to the not-evenclose-to-perfect world in which we all live. Here, observational studies often report on a variety of endpoints that include unconfirmed influenza-like illness, pneumonia hospitalization Corresponding author. National Institute of Allergy and Infectious Diseases, National Institutes of Health, 6700B Rockledge Drive, room 1107, Bethesda 20892, Maryland, USA. E-mail: lsimonsen@niaid.nih.gov Laboratory of Infectious Diseases, National Institute of Allergy and Infectious Diseases, National Institutes of Health, Bethesda, Maryland, USA. The online version of this article has been published under an open access model. Users are entitled to use, reproduce, disseminate, or display the open access version of this article for non-commercial purposes provided that: the original authorship is properly and fully attributed; the Journal and Oxford University Press are attributed as the original place of publication with the correct citation details given; if an article is subsequently reproduced or disseminated not in its entirety but only in part or as a derivative work this must be clearly indicated. For commercial re-use, please contact journals.permissions@oxfordjournals.org Published by Oxford University Press on behalf of the International Epidemiological Association. The Author 2007; all rights reserved. International Journal of Epidemiology 2007;1–2 doi:10.1093/ije/dym084