A blinded evaluation of a clinical decision-support system at a regional cancer care center.

Abstract

6553 Background: Clinical decision-support systems (CDSS) such as Watson for Oncology (WFO) may reduce treatment variation in oncology, provided options offered by the system are at least as acceptable as expert, evidence-based options. Deviation from expert consensus in practice is not well documented. In this blinded study, WFO therapeutic options and treatment decisions made by individual oncologists at Bumrungrad International Hospital (BIH) were evaluated by expert panel. Methods: Treatments selected by BIH that were labeled as either “for consideration” or “not recommended” by WFO were evaluated by a panel of 3 oncologists in 2018. The panel evaluated WFO options and previous BIH treatments for prospective cases from 2016-2018, blinded to the source of treatment option. Consensus of panel rated treatment pairs as: identical; both acceptable and roughly equivalent; both acceptable, but one preferred; one is acceptable and the other, unacceptable; neither is acceptable. The results of 321 treatment choices for breast, lung, colon and rectal cancers were analyzed, and McNemar’s test, a modified pairwise chi-square, was applied to identify differences between BIH and WFO. Results: 71% of both BIH and WFO treatments across all 4 cancer types were considered acceptable or identical by the panel. In 18 cases (5.6%), WFO treatments were preferred; in 14 cases (4.4%), BIH cases were preferred. Unacceptable treatments by either BIH or WFO were identified in 15% and 23% of treatments, respectively. Statistical analysis of treatment pairs revealed no significant difference between BIH and WFO treatments for breast, colon and rectal cancer. Treatment for lung cancer differed significantly ( p = 0.004); in 6% of cases, WFO was unacceptable and BIH acceptable; in 1% of cases, BIH was unacceptable and WfO was acceptable. Conclusions: This study is one of the first to compare therapeutic options from CDSS to treatment decisions made in practice, evaluated in a blinded fashion by an expert panel. 71% of treatments suggested by WFO CDSS were as acceptable as those selected by clinicians at the point of care, and some were considered superior. Decisions made in practice were unacceptable to the panel in 15% of cases, suggesting a role for CDSS.