A large language model-based clinical decision support system for syncope recognition in the emergency department: A framework for clinical workflow integration

Abstract

Differentiation of syncope from transient loss of consciousness can be challenging in the emergency department (ED). Natural Language Processing (NLP) enables the analysis of free text in the electronic medical records (EMR).The present paper aimed to develop a large language models (LLM) for syncope recognition in the ED and proposed a framework for model integration within the clinical workflow.Two models, based on both the Italian and Multilingual Bidirectional Encoder Representations from Transformers (BERT) language model, were developed using consecutive EMRs. The “triage” model was only based on notes contained in the “triage” section of the EMR. The “anamnesis” model added data contained in the “medical history” section. Interpretation and calibration plots were generated.The Italian and Multi BERT models were developed and tested on both 15,098 and 15,222 EMRs, respectively. The triage model had an AUC of 0·95 for the Italian BERT and 0·94 for the Multi BERT. The anamnesis model had an AUC of 0·98 for the Italian BERT and 0·97 for Multi BERT. The LLM identified syncope when not explicitly mentioned in the EMR and also recognized common prodromal symptoms preceding syncope.Both models identified syncope patients in the ED with a high discriminative capability from nurses and doctors’ notes, thus potentially acting as a tool helping physicians to differentiate syncope from others transient loss of consciousness.