Constructing adverse event timelines for patients receiving CAR-T therapy using large language models.

Abstract

2555 Background: Chimeric antigen receptor T (CAR-T) therapy is associated with a high risk of severe adverse events often only detailed in clinical notes. Monitoring them demands significant time and effort for manual chart review. Recent developments in large language modeling (LLMs) show promise for large-scale information extraction from clinical text. We performed a pilot study to evaluate the capability of the GPT-4 LLM to extract adverse events documented in the progress reports. Methods: We extracted progress notes within 30 days of any CAR-T administration from the UCSF deidentified clinical data warehouse. GPT-4, accessed through a HIPAA compliant Microsoft Azure Studio API, was used to extract CAR-T adverse events resulting in clinical intervention. A random sample of adverse events from 10% of patient notes were evaluated by a clinical reviewer (JG, PharmD). Topic modeling using BERTopic was used to cluster all adverse events to identify trends over time. Results: We identified 4183 clinical notes written within 30 days of CAR-T administration from 253 patients (39.1% women, 60.9% men). Mean age was 60.6 (SD:17.7). Manual validation of clinical notes from 10% of patients with CAR-T therapies (n=25) demonstrated that GPT4 was able to extract CAR-T related adverse events with 64% accuracy. We used BERTopic to cluster all extracted adverse events into 19 topics. Clusters with key terms “hyponatremia, leukocytosis, encephalopathy, toxicities, and neurologic” occurred most often (n=277), and primarily documented 12.9 days after CAR-T administration (Table). Conclusions: Although limited by use of de-identified data and absence of prompt engineering, this pilot supports the further investigation of LLMs for extraction of adverse events from unstructured clinical text. [Table: see text]