Estimating Treatment Effects Using Observational Data

Abstract

THE RANDOMIZED CLINICAL TRIAL (RCT) IS THE ideal method for measuring treatment effects. Participants in clinical trials are randomly assigned to a treatment or control group. Randomization reduces biases by making treatment and control groups “equal with respect to all features,” except the treatment assignment. When randomization is performed correctly, differences in efficacy found by statistical comparisons can be attributed to the difference between the treatment and control. However, the RCT does not necessarily provide the final answer to treatment effectiveness, as there are many restrictions that limit generalizability. For example, RCTs are often restricted to patients with limited disease, comorbidity, and concomitant medications. Thus, RCTs generally demonstrate efficacy rather than effectiveness, where efficacy is the treatment effect under the restricted conditions of the RCT and effectiveness is the treatment effect under the conditions of usual practice. Observational, nonrandomized studies have a role when RCTs are not available, and, even when RCTs are available, to quantify effectiveness and other real world experiences. A contemporary example of this is the evaluation of drugeluting stents, for which RCTs have demonstrated shortterm efficacy for relatively healthy patients and observational studies are beginning to address long-term effectiveness and safety problems and use of clopidogrel in a broader array of patients. There are many approaches for making statistical inferences from observational data. Some approaches focus on study design, others on statistical techniques. However, even with the best of designs, observational studies, unlike the RCTs, do not automatically control for selection biases. Therefore, statistical methods involving matching, stratification, and/or covariance adjustment are needed. Lack of randomization in observational studies may result in large differences on the observed (and unobserved) participant characteristics between the treatment and control groups. These differences can lead to biased estimates of treatment effects. The goal of the statistical techniques that focus on observational data is to create an analysis that resembles what would occur had the treatment been randomly assigned. In RCTs, the balance is achieved on participant characteristics that occur before the treatment is administered. The success of randomization in creating balance can be assessed before any outcome measurements are taken. Therefore, in observational studies, the first goal of a statistical technique is to create balance between treatments on characteristics that are assessed before the actual treatment is administered. Once this balance has been achieved, outcome measurements can be ascertained and compared between groups. In practice, this goal is often difficult to achieve because the data available for observational studies usually contain measured patient characteristics that are obtained before, during, and after treatment administration, and it is often difficult to determine exactly which patient characteristics are pretreatment or not. Furthermore, there frequently are unmeasured characteristics that are not available, inadequately measured, or unknown. Thus, the statistical methods in observational studies need first to be judged based on their performance in creating a balance on background characteristics between treated and control groups and the impact of outcome data should play no role in this assessment. To adjust for pretreatment imbalances, 2 statistical approaches often used are analysis of covariance methods and propensity score methods. These 2 methods complement each other and generally should be used together rather than choosing between one or the other. Analysis of covariance refers here to standard statistical analyses that produce estimates of treatment effects adjusted for background characteristics (covariates), which are included explicitly in a statistical (regression) model. With observational studies, this technique can produce biased estimates of treatment effects if there is extreme imbalance of the background characteristics and/or the treatment effect