Calibrating predictive model estimates in a distributed network of patient data

Abstract

BackgroundProtecting the privacy of patient data is an important issue. Patient data are typically protected in local health systems, but this makes integration of data from different healthcare systems difficult. To build high-performance predictive models, a large number of samples are needed, and performance measures such as calibration and discrimination are essential. While distributed algorithms for building models and measuring discrimination have been published, distributed algorithms to measure calibration and recalibrate models have not been proposed. ObjectiveRecalibration models have been shown to improve calibration, but they have not been proposed for data that are distributed in various health systems, or “sites”. Our goal is to measure calibration performance and build a global recalibration model using data from multiple health systems, without sharing patient-level data. Materials and MethodsWe developed a distributed smooth isotonic regression recalibration model and extended established calibration measures, such as Hosmer-Lemeshow Tests, Expected Calibration Error, and Maximum Calibration Error in a distributed manner. ResultsExperiments on both simulated and clinical data were conducted, and the recalibration results produced by a traditional (ie, centralized) versus a distributed smooth isotonic regression were compared. The results were exactly the same. DiscussionOur algorithms demonstrated that calibration can be improved and measured in a distributed manner while protecting data privacy, albeit at some cost in terms of computational efficiency. It also gives researchers who may have too few instances in their own institutions a method to construct robust recalibration models. ConclusionPreserving data privacy and improving model calibration are both important to advancing predictive analysis in clinical informatics. The algorithms alleviate the difficulties in model building across sites.