Validation of article analysis by ChatGPT: Using a cohort of patients with EGFR mutation-positive non-small cell lung cancer treated with first-line osimertinib.

Abstract

e20542 Background: The rapid advancement of large language models like ChatGPT has shown potential in various fields, including medicine. However, their applicability in clinical practice, especially in complex medical knowledge interpretation, remains unclear. This study developed a framework for article analysis, retrieval of additional information, and knowledge updating using ChatGPT and tested its clinical validity by comparing it to clinical data. Methods: To simplify the prompt to ChatGPT by selecting monotherapy, patients with EGFR mutation-positive non-small cell lung cancer (NSCLC) who had received 1st -line treatment with Osimertinib, were included in this study. ChatGPT analyzed published FLAURA trial data and developed a scoring system for predicting treatment efficacies for comparison with clinical data. Treatment efficacies were evaluated by using progression-free survival (PFS) and overall survival (OS). ChatGPT searched for papers needed to update the scoring system and extracted 5 papers. To reduce the bias of the search, the extraction of papers by ChatGPT was repeated 10 times and the top 5 papers were selected. The revised scoring system, incorporating insights from these 5 papers, was applied to data from Shizuoka Cancer Center (1st-line osimertinib initiated during 2018-2022.3). The clinical validity of analyses by ChatGPT was evaluated in terms of whether the scoring system reflects clinical practice. Analysis with ChatGPT was performed using GPT4 and natural language prompts. Results: Based on the FLAURA study, ChatGPT created a scoring system consisting of the following items: EGFR mutation type, presence of CNS metastasis, smoking history, sex, and performance status (PS). ChatGPT extracted two prospective studies and three retrospective studies. The scoring system, updated by ChatGPT included EGFR mutation type: exon 19 del /L858R (3/1point), sex: female/male (2/1), smoking history: never-smoker/smoker (2/1), CNS metastases: without/with (2/1), PS 0/1/2/3/4 (2/1/0/-1/-2), age: < 75/≥ 75 years (2/1) PD-L1 expression < 50%/≥ 50% (2/0); liver metastasis, without/with (2/0). Group A showing better efficacies to treatment was defined as patients with a cutoff value of 11 or higher. The scoring system was fitted to validate cohort (140 patients without missing scoring system items). Group A (n = 98, ≥11points) outperformed Group B (n = 42, < 11) in both PFS and OS (medianPFS 22.6 months vs. 9.7 months HR 0.32, 95% CI 0.20-0.50, p < 0.0001, medianOS 54.4 months vs. 21.7 months HR 0.28, 95%CI 0.16-0.49, p < 0.0001). Conclusions: The scoring system created by using ChatGPT reflected the efficacies in a cohort of EGFR-positive NSCLC patients treated with 1st-line osimertinib. The framework for article analysis, retrieval of additional information, and knowledge update using ChatGPT is potentially applicable to other cohorts.