Generative AI enhanced with NCCN clinical practice guidelines for clinical decision support: A case study on bone cancer.

Abstract

e13623 Background: Bone cancer is a complex and challenging disease to diagnose and treat in clinical practice. Recently, generative AI, especially large language models (LLMs), has demonstrated potential as a decision support tool for cancer. However, most implementations have overlooked the integration of available cancer guidelines, such as the NCCN Bone Cancer Guidelines, in fine-tuning the outputs of generative AI models. Incorporating these guidelines into LLMs presents an opportunity to harness the extensive clinical knowledge they contain and improve the decision-support capabilities of the model. Methods: In this study, the aim is to enhance the LLM with cancer clinical guidelines to enable accurate medical decisions and personalized treatment recommendations. Therefore, we introduce a novel method for incorporating the NCCN Bone Cancer Guidelines into LLMs using a Binary Decision Tree (BDT) approach. The approach involves constructing a BDT based on NCCN Bone Cancer Guidelines, where internal nodes represent decision points from the Guidelines, and leaf node signify final treatment suggestions. Then the LLM makes decision at each internal node, considering a given patient's characteristics, and guides toward a treatment recommendation in the leaf node. To assess the efficacy of Guideline-enhanced LLMs, an oncologist from our team created 11 hypothetical osteosarcoma patients’ medical progress notes. Each note contains their demographics, medical history, current illness, physical exams, diagnostic tests. We tested three LLMs in the implementation (GPT-4, GPT-3.5, and PaLM 2) and compared the LLM-generated treatment recommendations with the gold standard treatment across four runs with different random seeds (random seeds is a setting to control the LLM outputs). The results are reported as the average of four runs. The original LLMs are used as baseline methods for comparison. Results: The table below provides a comparison between the performance of original LLMs and those augmented with cancer guidelines for osteosarcoma treatment recommendations. We can observe that the PaLM 2 model demonstrated superior performance compared to its counterparts, underscoring the effectiveness of integrating cancer guidelines into LLMs for decision support. Conclusions: The clinical decision support capabilities of the LLMs are promising when enhanced by NCCN Bone Cancer Guidelines using our approach. To fully exhibit the potential of our proposed method as a clinical decision support tool, further investigation into other subtypes of bone cancer should be conducted in the future study. [Table: see text]