Impact of learning program on treatment recommendations by molecular tumor boards and an artificial intelligence-based annotation system: A prospective study.

Abstract

11032 Background: Treatment recommendations (TRs) by molecular tumor boards (MTBs) based on comprehensive genomic profiling tests are crucial for enrolling cancer patients in genotype-matched trials. Our previous study of Japanese designated ‘Core’ hospitals using simulated cases showed that TRs for biomarkers with high evidence levels (ELs) were highly concordant between central consensus and MTBs, whereas those with low ELs showed low concordance, indicating that a learning program (LP) about obtaining information on matched trials, particularly for those with low ELs, is needed to improve the quality of TRs by MTBs (Kage H. ESMO 2021). Therefore, we conducted a nationwide prospective study to investigate the clinical utility of an LP for MTBs in designated ‘Hub’ hospitals and explore the efficacy of an artificial intelligence (AI)-based annotation system. Methods: Fifty simulated cases were randomly divided into 2 groups (pre- and post-tests) by the central statistician, stratified by high vs. low ELs. Each MTB at ‘Hub’ hospitals (group), treating physicians at ‘Core’ hospitals (individual), and an AI company were eligible. Each participant made TRs for the first 25 cases, then participated in the LP, and made TRs for the second 25 cases. The online LP shared the methodology of making the optimal TRs and showed central consensus TRs for simulated cases. The primary endpoint was the proportion of MTBs that met prespecified ‘accreditation’ criteria for the second 25 cases: > 90% concordance for high ELs and > 40% for low ELs. Expected and threshold proportions of ‘accreditation’ MTBs were 50% and 20%, respectively, and the planned sample size was 24 MTBs. The improvements in concordance rates in TRs on post-tests were key secondary endpoints. The concordance rate of TRs with the AI system was an exploratory endpoint. Results: From May to December 2021, 47 participants applied, and 42 (27 MTBs, 14 individuals, and an AI company) were eligible. The primary ‘accreditation’ rate of MTBs was 55.6% (95% CI, 35.3-74.5%, p < 0.001), and the ‘accreditation’ rate of individuals was 35.7% (95% CI, 12.8-64.9%, p = 0.17). TR concordance rates improved significantly, from 57.5% (95% CI, 51.7-63.1%) to 65.7% (95% CI, 59.2-71.6%) [odds ratio (OR): 1.23, p = 0.04]. A prespecified subgroup analysis showed an improved concordance rate for biomarkers with low ELs for MTBs (OR, 1.32: 95% CI, 1.00-1.73), but not for individuals (OR, 1.11: 95% CI, 0.75-1.63). TR concordance rates with the AI system were 80% (95% CI, 60.0-91.4%) and 84% (95% CI, 64.3-93.9%) for pre- and post-tests, respectively. Conclusions: This LP significantly improved the quality of TRs by MTBs, potentially leading to providing more matched trials to cancer patients. TRs by the AI system showed higher concordance with central consensus TRs than MTBs, suggesting the clinical utility of AI-based TRs over those by MTBs.