Validating a Budget Impact Model Using Payer Insight and Claims Data: A Framework and Case Study

Abstract

There is a paucity of studies validating budget impact models. The lack of such studies may contribute to the underuse of budget impact models by payers in formulary decision making. To assess the face validity, internal verification, and predictive validity of a previously published model that assessed the budgetary impact of antidiabetic formulary changes. 4 experts with diverse backgrounds were selected and asked questions regarding the face validity of the structure/conceptual model, input data, and results from the budget impact model. To assess internal verification, structured "walk-throughs," unit tests, extreme condition tests, traces, replication tests, and double programming techniques were used. The predictive validity of the model was evaluated by comparing the predicted and realized budget using mean absolute scaled error. "Realized" budgetary impact of the formulary changes was calculated by taking the difference between realized budget in the year after the formulary changes and the budget had there been no formulary changes (i.e., the counterfactual). The counterfactual budget was modeled using the best fit autoregressive integrated moving average model. When assessing the face validity of the model, the 4 experts brought up issues such as how to incorporate other health insurance, recent policy changes, cost inflation, and potential impacts on insulin use. The 6 internal verification techniques caught mistakes in equations, missing data, and misclassified data. The realized budget was found to be lower than the predicted budget, with 13% error and an absolute scaled error of 2.60. After removing the model assumption that past utilization trends would continue, the model's predictive accuracy improved (the absolute scaled error dropped below 1 to 0.48). The "realized" budgetary impact was found to be greater than the predicted budgetary impact, largely because of lower-than-expected utilization. The budget impact model overpredicted utilization in the year after the formulary changes. Discoveries through the validation process improved the accuracy and transparency of the model. This project was supported by grant number F32HS024857 from the Agency for Healthcare Research and Quality (AHRQ). The content is solely the responsibility of the authors and does not necessarily represent the official views of AHRQ. AHRQ had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; or design to submit the manuscript for publication. The findings discussed in this manuscript represent the views of the authors and do not necessarily reflect the views of the Department of Defense, the Defense Health Agency, nor the Departments of the Army, Navy, and Air Force. Hung reports a grant from the AHRQ, during the conduct of the study, and personal fees from CVS Health and BlueCross BlueShield Association, outside the submitted work. Mullins reports grants and personal fees from Bayer and Pfizer and personal fees from Boehringer Ingelheim, Janssen/J&J, Regeneron, and Sanofi-Aventis, outside the submitted work. Mullins, Slejko, and Shaya are employed by the University of Maryland School of Pharmacy. Haines and Lugo have nothing to disclose. Part of this content was previously presented as a poster at the 2017 AMCP Managed Care & Specialty Pharmacy Annual Meeting; March 27-30, 2017; Denver, CO, and as poster and oral presentations at the 2017 AMCP Nexus Meeting; October 16-19, 2017; Dallas, TX. Part of this content was published as Hung's PhD dissertation.