How robust are reference pricing studies on outpatient medical procedures? Three different preprocessing techniques applied to difference-in differences.

Abstract

The evaluation of policies that are not randomly assigned on outcomes generated by nonlinear data generating processes often requires modeling assumptions for which there is little theoretical guidance. This paper revisits previously published difference-in-differences results of an important example, the introduction of reference pricing to common outpatient procedures, to assess the robustness of the estimated impacts by using different matching, and reweighting techniques to preprocess the data. These techniques improve covariate balance and reduce model dependence. Specifically, we examine the robustness of the effect of reference pricing on patient site-of-care choice, total expenditures, and complication rates. We apply three preprocessing methods: propensity score reweighting, exact matching, and genetic matching. Propensity score reweighting is a technique for achieving covariate balance but does not balance higher-order moments and may lead to bias and inefficiency in estimating treatment effects in the context of nonlinear data generating processes. In contrast, exact matching and genetic matching are designed to balance higher-order moments. We find that although the use of the preprocessing techniques is a valuable robustness check showing that some results are sensitive to the method used, the three approaches generally yield results that do not statistically differ from the published results.