Limitations in the use of automated mental status detection for clinical decision support

Abstract

BackgroundClinical decision support (CDS) tools improve adherence to evidence-based practices but are dependent upon data quality in the electronic health record (EHR). Mental status is an integral component of many risk stratification scores, but it is not known whether EHR-measures of altered mental status are reliable. The Glasgow Coma Scale (GCS) is a measure of altered mentation that is widely adopted and entered in the EHR in structured format. We sought to determine the accuracy GCS < 15 as an EHR-measure of altered mentation compared to ED provider documentation. MethodsIn patients presenting to an academic Emergency Department (ED) with pneumonia we abstracted GCS values entered by nurses during routine care and in a randomly selected subset manually reviewed provider documentation for evidence of altered mental status. We defined eConfusion as present if GCS < 15 at any point during the ED encounter. We then calculated the CURB-65 score and corresponding suggested disposition using each method. Performance of eConfusion and corresponding CURB-65 compared to manual versions was measured using agreement (Cohen’s K), sensitivity, and specificity. ResultsAmong 300 randomly selected encounters, 47 (16 %) had eConfusion present and 46 (15 %) had evidence of altered mental status in provider documentation with Cohen’s K 0.73. eConfusion had 78 % sensitivity and 96 % specificity for provider documented altered mental status. When input into CURB-65 to recommend inpatient disposition, eConfusion had 95 % sensitivity, and recommended discordant disposition for 3 %. ConclusionsThere was modest agreement between eConfusion and provider documentation of altered mental status. eConfusion had good specificity but low sensitivity which resulted in under-estimation of the CURB-65 score and occasional inappropriate disposition recommendations compared to provider documentation. These data do not support the use of GCS as a measure for altered mentation for use in CDS tools in the ED.