Abstract P4-14-05: A systematic review of concordance studies using Watson for Oncology (WfO) to support breast cancer treatment decisions: A four-year global experience

Abstract

Abstract Background: Watson for Oncology (WfO) is an artificial intelligence (AI)-based clinical decision-support system (CDSS) that presents personalized therapeutic options to support cancer-treating physicians in making treatment decisions for cancer patients. WfO was released in 2015, and it has been deployed in over 200 institutions across the world. Many academic centers have examined WfO’s performance by measuring concordance between WfO therapeutic options and treatment recommendations by multidisciplinary tumor boards (MTBs) or individual clinicians (ICs). This study systematically reviewed the results of such concordance studies for breast cancer. Methods: We conducted a review of the WfO publication database and a PubMed search to identify WfO concordance studies in breast cancer patients, published from 01/01/2015 to 06/30/2019. Studies were excluded if they measured concordance for multiple cancer types but did not include individual concordance for breast cancer. Concordance was defined as agreement between WfO “Recommended” and “For Consideration” treatment options and treatments prescribed by MTBs or ICs. Mean concordance rates were calculated as an average, weighted by the number of patients in each study. Concordance rates between MTBs and ICs were compared with z-test of two proportions. Subgroup analyses for larger studies were summarized. Results: Table 1 presents the results of nine identified breast cancer concordance studies (4,427 patients) from China, India, and Thailand. Five studies (1,528 patients) determined concordance with MTB and 4 (2,899 patients) with ICs. WfO treatment options were compared to historical treatment recommendations by MTBs and ICs. Mean concordance for all studies was 70.8% (range 55 - 98%). Mean concordance with MTBs of 90.7% (range 79 - 98%) was significantly higher than the concordance between WfO and ICs of 59.9% (range 55-76%) p<0.0001. Table 1Study/LocationNumber of patientsConcordance Multidisciplinary Tumor Board Studies (MTBs)Zhang XC, et al. Ann Oncol 2017;28:x170 / China11979%Yue L, Yang L. Ann Oncol 2017;28:x162 / China3198%Somashekhar SP, et al. Ann Oncol 2018 1; 29(2):418-423 / India63893%Zhou N, et al. The Oncologist 2018;23:1-8 / China12082%Somashekhar SP, et al. J Clin Oncol 2019; 37 (suppl; abstr 6533) / India62092%MTB subtotals152890.7%Individual Clinicians Studies (ICs)Suwanvecho S, et al. J Clin Oncol 2017;35 (suppl; abstr 6589) / Thailand21176%Jiang Z, et al. J Clin Oncol 2018;36 (suppl; abstr 18566) / China1,99755%Suwanrusme H, et al. J Clin Oncol 2018;36 (suppl; abstr 18584) / Thailand26470%32672%Suwanvecho S et al. J Clin Oncol 2019; 37 (suppl; abstr 6553) / Thailand10160%ICs subtotals289959.9% Mean concordance in China, Thailand and India were 78.5%, 69.5%, and 92.5%, respectively. A large IC study from China including 1,997 patients found concordance in triple negative, high risk non-metastatic, and metastatic breast cancers of 69%, 66%, and 50%, respectively. A MTB study from China in 120 patients found concordance for luminal A, luminal B and triple negative breast cancers of 63%, 87% and 79%, respectively. Reported reasons for discordance varied among studies and included locally unavailable treatments, individual physician or MTB management preferences, and age older than 75 years. Conclusions Overall concordance between the WfO therapeutic options and decisions of both MTBs and ICs was high, demonstrating performance comparable with experts across the world. Higher concordance was observed between WfO and MTBs versus WfO and ICs, likely reflecting the multidisciplinary expertise having greater agreement with evidence and guideline-based recommendations of WfO than decisions of individual clinicians. This finding illustrates a role for clinical decision support in practice. Concordance varied across countries, reflecting the need for localization to address regional differences in practice. Citation Format: Yull Arriaga, Rezzan Hekmat, Karlis Draulis, Suwei Wang, Winnie Felix, Irene Dankwa-mullan, Kyu Rhee, Gretchen Jackson. A systematic review of concordance studies using Watson for Oncology (WfO) to support breast cancer treatment decisions: A four-year global experience [abstract]. In: Proceedings of the 2019 San Antonio Breast Cancer Symposium; 2019 Dec 10-14; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2020;80(4 Suppl):Abstract nr P4-14-05.