Delta Studies: Expanding the Concept of Deviance Studies to Design More Effective Improvement Interventions.

Abstract

The effects of improvement (implementation and de-implementation) interventions are often modest. Although positive and negative deviance studies have been extensively used in improvement science and quality improvement efforts, conceptual and methodological innovations are needed to improve our ability to use information about variation in quality to design more effective interventions. We describe a novel mixed methods extension of the deviance study we term "delta studies." Delta studies seek to quantitatively identify sites that have recently changed from low performers to high performers, or vice versa, in order to qualitatively learn about active strategies that produced recent change, challenges change agents faced and how they overcame them, and where applicable, the causes of recent deterioration in performance-information intended to inform the design of improvement interventions for deployment in low performing sites. We provide examples of lessons learned from this method that may have been missed with traditional positive or negative deviance designs. Considerations for quantitatively identifying delta sites are described including which quality metrics to track, over what timeframe to observe change, how to account for reliability of observed change, consideration of patient volume and initial performance as implementation context factors, and how to define clinically meaningful change. Methods to adapt qualitative protocols by integrating quantitative information about change in performance are also presented. We provide sample data and R code that can be used to graphically display distributions of initial status, change, and volume that are essential to delta studies. Patients and facilities of the US Veterans Health Administration. As an example, we discuss what decisions we made regarding the delta study design considerations in a funded study of low-value preoperative testing. The method helped us find sites that had recently reduced the burden of low-value testing, and learn about the strategies they employed and challenges they faced. The delta study concept is a promising mixed methods innovation to efficiently and effectively identify improvement strategies and other factors that have actually produced change in real-world settings.