From Methods to Policy: Observational studies versus randomized trials: squaring off

Abstract

A recent study, published in the New England Journal of Medicine, highlights the challenges involved in attempting to balance treatment groups in observational studies to approximate randomized trials [1]. Specifically, the study focuses attention of the role of propensity scores (PS) in compensating for the lack of randomization. The study sought to compare percutaneous coronary intervention (PCI) to coronary artery bypass graft (CABG) in patients 65 years of age or older who had twoor three-vessel coronary artery disease. Of the nearly 2 million unique patients, 103,549 who underwent PCI and 86,244 who had a CABG were included in the analysis. The data, derived from claims records, represented a major national undertaking by the American College of Cardiology Foundation and the Society for Thoracic Surgeons in the USA. PS with inverse probability weighting was used to match patients in each treatment group by specific characteristics of patients and hospitals. At 4 years, in the matched group, a 4% lower mortality was observed for CABG patients compared with patients who underwent PCI. The study relies on the use of appropriate potential confounding variables in the analysis. With a 4% difference between groups, although an individual variable may not have caused enough confounding to reduce the difference, as the authors point out in a simulation analysis, various combinations of these variables could well have caused selection imbalance that would have influenced the results. Summary measures have been developed and tested for, for example, atrial fibrillation (CHADS 2 ), and for overall health [2], but were not used in this study. The sheer size of the database could be somewhat misleading about the generalizability of the study findings. Only a relatively small percentage of the total sample was matched on PS. In the subset well matched on PS, important clinical risk variables such as the extent of coronary disease, the presence of chronic coronary occlusions and patient ‘frailty’, among others, were not included. The use of PS is being advanced as an approach to balance groups in comparative effectiveness research when randomization is not possible or desirable. Researchers feel great pressure to employ registries and previously existing databases to conduct observational studies. However, studies must be methodologically rigorous, and include efforts to address confounding by indication. PS, with stratification, regression and adjustment with inverse weighing based on PS, represent an effort to reduce the threats of confounding to the point where observational studies can make useful contributions to effective clinical practice. Only judgment and experience will tell us whether such efforts methodologically advance clinical research. However, as Robinson has pointed out in an earlier issue of Journal of Comparative Effectiveness Research [3], that judgment could be made considerably easier if PS, or another suitable strategy, was integrated into a robust series of study designs to address a clinical issue rather than being employed as a post hoc statistical technique used to compensate for study inadequacies. This shift in thinking places “There is great momentum to create a ‘learning healthcare’ environment, where the data from each patient, included in routinely collected databases, can be used to estimate comparative effectiveness of drugs, procedures, images and healthcare systems.”