Large language models to evaluate racial discrepancies in performance status assignment.

Abstract

1561 Background: Oncologic treatment eligibility relies heavily on “Performance Status” (PS), a subjective gauge of a patient's overall health. Despite frameworks aimed at quantifying PS, these assessments remain prone to potential bias. A systematic evaluation of discrepancies in language between ECOG PS and race has yet to be conducted. Large language models (LLMs) have the ability to synthesize text and may enable an assessment of the relationship between race, note text, and physician-documented PS. We hypothesize that LLMs can quantify these relationships to understand potential inconsistencies in ECOG PS. Methods: In our single-institution cohort study, we examined patients from medical or radiation oncology clinics between January 2012 to December 2023 with documented ECOG PS. PS was extracted from clinical notes and redacted from downstream analyses. Notes were categorized by patient-reported race (Asian, Black, White, Other). 1,500 matched subjective assessments (from oncologist clinical notes) and PS were randomly sampled for each race, except for Black patients, where only 640 entries were available. We divided training and test sets using an 80-20 ratio across each race. Using the training cohort, we trained race-specific models to assign ECOG PS based on subjective assessment, using two LLMs: UCSFBERT (BERT pretrained on UCSF data) and RoBERTa. In the hold-out test sets, we applied each model across all races to assign ECOG PS based on subjective assessments. These were evaluated by micro-F1 (model accuracy) and Multiway ANOVA to compute model-specific p-values. Results: 13.6% of patients had a documented ECOG 2+, with variation across races (Asian: 16.1%, Black: 17.7%, White: 7.7%, Other: 15.3%). Models tended to assign Black patients with ECOG 0-1 with worse performance status (ECOG 2+). Other than RoBERTa models trained on Asian and White cohorts, all models trained on a specific race showed disparate classification results when applied to other races (out-of-domain), highlighting potential biases in LLM models trained where a specific race predominates (Table; ANOVA). Conclusions: This study demonstrates a novel way to use LLM to assess discordance within physician documentation and performance status assignment. It underscores the need for incorporating diverse demographic data when using LLM in medical contexts. This research has potential to dissect how bias may be propagated within physician practice and provide insight into known disparities in clinical trial enrollment and standard of care implementation.[Table: see text]