Investigating Applications of Prognostic Scores to Balancing Weights: Evidence from an Evaluation of Accountable Care Organizations

Abstract

Research ObjectivePropensity score methods, based on the predicted probability of treatment, are ubiquitous in health services research. Over the past decade, an increasing amount of research has investigated applications of prognostic scores: the predicted probability of the study outcome of interest (e.g., illness, mortality). However, this research has focused on the use of prognostic scores in matching, either in conjunction with, or in place of, more commonly‐implemented propensity score matching approaches. There has been little, if any, research focused on the use of prognostic scores with other analytic strategies such as weighting or subclassification typically associated with propensity scores. Prior research has also focused on prognostic scores for binary outcomes. This study presents several analytic strategies based on a continuous pseudo‐prognostic score (P‐PS) and assesses their practical value relative to a public evaluation of a Medicare value‐based payment model.Study DesignWe used Medicare claims and enrollment data covering all 41 Medicare accountable care organizations (ACOs) that participated in the ACO Investment Model (AIM): the same data used to evaluate the first AIM performance year. The evaluation's primary outcome of interest was total Medicare spending, per‐beneficiary‐per‐month. We estimated a regression model for total spending using each ACO's comparison group and predicted the P‐PS (i.e., predicted spending) for intervention and comparison beneficiaries using the model's parameters.While the original evaluation used entropy‐balancing weights to balance all covariates between intervention and comparison beneficiaries within each ACO, we calculated three types of enhanced entropy‐balance weights based on the P‐PS. We balanced intervention and comparison beneficiaries on all covariates and P‐PS, and then incorporated a quadratic P‐PS as well. Lastly, we classified beneficiaries into five quintiles of P‐PS, and separately balanced covariates within each quintile.We assessed the effect of the enhanced weights on the magnitude and precision of corresponding impact estimates, as well as the validity of the parallel trend assumption underlying the difference‐in‐differences (DID) evaluation design, relative to the original weights.Population StudiedBeneficiaries with fee‐for‐service Medicare attributed to ACOs in AIM and corresponding comparison beneficiaries eligible for attribution in 2013–2016.Principal FindingsIncorporating P‐PS directly into the weights did not meaningfully affect estimated differences in baseline parallel trends or impact estimates. However, across the 41 ACOs, weighting by covariates within PPS quintiles reduced average absolute baseline differences by 14.7% at the mean and 21.2% at the median. These differences shifted the mean impact estimate by 6.7%, while increasing the mean standard error of the impact estimates by 2.3%.ConclusionsBalancing covariates within P‐PS quintiles improved the validity of DID estimates by substantially reducing differences in baseline trends, and meaningfully reduced bias in the subsequent impact estimates. Bias reduction was greater than the loss in precision. Incorporating P‐PS directly into the weights did not meaningfully affect baseline or impact estimates.Implications for Policy or PracticeBalancing covariates within prognostic score quintiles shows promise for reducing bias in observational study designs. Unlike matching‐based prognostic score approaches, this method can be implemented when outcomes of interest are continuous, and does not require discarding unmatched data.Primary Funding SourceCenters for Medicare and Medicaid Services.