Time-series and risk-adjusted control charts

Abstract

Presentation There are several statistical process control (SPC) methods used in industry that can be applied in healthcare. However, as noted in the earlier discussion by Toulany et al. regarding quasi-experimental designs for quality improvement research, several considerations must be taken into account when adapting these methods for the complex, high-risk healthcare arena. Industrial methods should be adjusted for (a) heterogeneity at the patient level, including illness type, individualized care, and demographics, (b) heterogeneity at the process level, including geographical and longitudinal clinical care variation, (c) lack of pre-existing standards of comparison for new products or processes, and (d) the critical difference between statistical variation and acceptable clinical risk. Potential methods for successful adaptation of industrial SPC methods for healthcare monitoring and improvement include (a) converting periodic data into cumulative charts to increase detection of trends and (b) addressing heterogeneity through risk adjustment, using a prediction model or propensity score matching. These adjustments tend to inflate Type I errors, however, due to repeated measurements. Thus, the Sequential Probability Ratio Testing (SPRT) method may be of particular use [1]. SPRT uses the more commonly available retrospective control data, accounts for repeated measurements, utilizes risk adjustment, and incorporates both alpha and beta error into the formal framework [2,3]. The upper control limit is the desired odds ratio, as determined by the hypothesis. Industrial SPC methods assume process homogeneity and that the outcome rate from the population establishes the threshold for detecting changes in the process. Using these techniques to analyze processes in healthcare often requires addressing the risk and complexities inherent in healthcare in order to obtain meaningful results. As with any data-driven project, the clinical question and limitations of the available data drive the selection of the patient cohort, SPC method, risk adjustment framework, alerting thresholds, and the interpretation of clinical significance. However, regardless of the SPC method used and the risk-adjustment framework, it is important to realize that performance of the risk adjustment model drives the overall result; thus understanding the strengths and weaknesses of each particular model is critical to clinical interpretations. In addition, detecting adverse outcomes over a long period requires recalibrating the model over time to adjust for systematic changes in clinical care. Finally, all signals detected using these methods require root cause analyses (RCA) and sensitivity analyses as they are hypothesis-generating, not confirming.