Comments on ‘The use of propensity scores and observational data to estimate randomized controlled trial generalizability bias’ by Taylor R. Pressler and Eloise E. Kaizar,Statistics in Medicine 2013

Abstract

It has been widely accepted that most randomized control trials (RCTs) include patient groups that are not a representative sample of the patients who will receive the intervention in daily practice [1–3]. This has raised concerns about the generalizability of RCT results. Recently, Pressler and Kaizar [4] enriched the discussion by asserting that the bias that results from a lack of generalizability can be quantified. They define two populations: the first consisting of subjects fulfilling the inclusion criteria of an RCT (in their notation group I) and the second group (group E) comprising subjects not meeting the inclusion criteria. They propose to estimate the treatment effects in both groups ( O .I / and O .E/, respectively) using nonrandomized (i.e., observational) data. Assuming equal amounts of confounding in both groups, Ǒ D O .I / O .E/ provides an unbiased estimate of how much treatment effect modification there exists between included and excluded subjects. If the interest is in estimating the ‘population average treatment effect’ (PATE), weighing Ǒ by the proportion of E among the total population of interest O D nE n provides an estimate O D O Ǒ of how much ‘generalizability bias’ is created by relying on O .I / to estimate the PATE. This weighing of Ǒ is necessary because if there is treatment effect modification between groups I and E, the PATE is dependent on the proportionate size of both groups. While we acknowledge the relevance of the approach suggested by Pressler and Kaizar, we wish to touch upon some concerns and discuss alternative strategies for exploring generalizability. First, Pressler and Kaizar fail to address why one would be interested in the treatment effect in group E. For example, if we explore the effectiveness of a new antihypertensive drug and E comprises subjects without hypertension, it seems illogical to try to estimate treatment effect modification between groups I and E. Second, using nonrandomized data to estimate O or Ǒ only results in an unbiased estimate if the amount of confounding is equal in both groups E and I. This assumption is not testable, as the authors acknowledge, and results in a problem encountered in virtually all nonrandomized studies, that