Evaluation of an automated method to assist with error detection in the ACCORD central laboratory

Abstract

Errors in clinical laboratory data are rare but their potential high cost both in terms of harm to the subject as well as diluted statistical power results in a significant workload for experts, who must review large volumes of data in the search for these errors. The current research objective is to develop and evaluate a method to assist in detecting potential errors in laboratory data for an interventional clinical trial, such as Action to Control Cardiovascular Risk in Diabetes, where treatment effects may be influenced by errors in the data. Utilizing data from a clinical trial investigating the effect of intensive glycemic control on major cardiovascular disease events, we constructed an algorithm that conducts probabilistic error detection called a 'Bayesian network'. Using a synthetic error model, errors were introduced into a testing dataset, and the Bayesian network's performance in identifying those errors was compared to laboratory experts. For each laboratory result we used the Bayesian network to compute the probability, the measured value was erroneous. This probability was then used to compute a receiver operating characteristic (ROC) curve. Three laboratory experts were recruited and took a survey consisting of 200 laboratory results. The task was to evaluate if these results were erroneous or not and to provide a confidence rating on a 6-point subjective probability scale. The Bayesian network's overall area under the ROC curve was calculated to be 0.79, whereas the three laboratory experts had areas under the ROC curve of 0.73, 0.73, and 0.72. Perfect error prediction and random guessing yield a ROC of 1.00 and 0.50, respectively. This difference in performance was statistically significant for all three experts. Human experts were also generally overconfident in their ability to detect errors. The model is, by design, specific to a novel intervention in a specific diabetic population and, therefore, the specific Bayesian network discussed may not generalize to other interventions and populations. In addition, the study is limited by the small number of expert eligible to complete the survey. The results of this study suggest that continuous Bayesian networks, suitably constructed, may serve as an effective tool to assist experts in the review of voluminous laboratory data by flagging unlikely results for more thorough review.