Characterizing research domain criteria symptoms among psychiatric inpatients using large language models

Abstract

We sought to characterize the ability of large language models to estimate NIMH Research Domain Criteria dimensions from narrative clinical notes of adult psychiatric inpatients, deriving estimate of overall burden of symptoms in each domain. We extracted consecutive admissions to a psychiatric inpatient unit between December 23, 2009 and September 27, 2015 from the electronic health records of a large academic medical center. Admission and discharge notes were scored with a HIPAA-compliant instance of a large language model (gpt-4–1106-preview). To examine convergent validity, the resulting estimates were correlated with those derived using an earlier method; for predictive validity, they were examined for association with length of hospitalization and probability of readmission. The cohort included 3619 individuals, 1779 female (49 %), 1840 male (51 %) with mean age 44 (SD=16.6). We identified modest correlations between LLM-derived RDoC scores and a previously validated scoring method, with Kendall’s tau between from.07 for arousal and 0.27 for positive and cognitive domains (p < .001 for all of these). For admission notes, greater scores on cognitive, sensorimotor, negative, and social domains were significantly associated with longer length of hospitalization in linear regression models including sociodemographic features (p < .01 for all of these); positive valence was associated with shorter hospitalization (p < .001). For discharge notes, social, arousal, and positive valence were associated with likelihood of readmission within 180 days in adjusted logistic regression models (p < .05 for social and arousal, p < .001 for positive valence). Overall, LLM-derived estimates of RDoC psychopathology demonstrated promising convergent and predictive validity, suggesting this approach may make real-world application of the RDoC framework more feasible.