A Weakly-Supervised Named Entity Recognition Machine Learning Approach for Emergency Medical Services Clinical Audit.

Han Wang,Shalini Arulanandam,Qin Xiang Ng,Angeline Tung,Joey Ai Meng Tay,Wesley Lok Kin Yeung,Mengling Feng,Marcus Eng Hock Ong,Davin Ryanputra

doi:10.3390/ijerph18157776

Abstract

Clinical performance audits are routinely performed in Emergency Medical Services (EMS) to ensure adherence to treatment protocols, to identify individual areas of weakness for remediation, and to discover systemic deficiencies to guide the development of the training syllabus. At present, these audits are performed by manual chart review, which is time-consuming and laborious. In this paper, we report a weakly-supervised machine learning approach to train a named entity recognition model that can be used for automatic EMS clinical audits. The dataset used in this study contained 58,898 unlabeled ambulance incidents encountered by the Singapore Civil Defence Force from 1st April 2019 to 30th June 2019. With only 5% labeled data, we successfully trained three different models to perform the NER task, achieving F1 scores of around 0.981 under entity type matching evaluation and around 0.976 under strict evaluation. The BiLSTM-CRF model was 1~2 orders of magnitude lighter and faster than our BERT-based models. Our proposed proof-of-concept approach may improve the efficiency of clinical audits and can also help with EMS database research. Further external validation of this approach is needed.

Highlights

Clinical performance audits are thought to be an important part of quality review and continuous quality improvement in healthcare systems and services [1,2]
In emergency medical services (EMS), one of the clinical audits that is conducted involves examining whether paramedics have performed the assessment and treatment steps following the standard operating procedures [3,4,5]
It is impractical to label a large amount of corpus to generate a dataset for fully supervised training. In this proof-of-concept study, we aimed to develop an Named entity recognition (NER) model on paramedic text reports for clinical audit

Summary

Introduction

Clinical performance audits are thought to be an important part of quality review and continuous quality improvement in healthcare systems and services [1,2]. In emergency medical services (EMS), one of the clinical audits that is conducted involves examining whether paramedics have performed the assessment and treatment steps following the standard operating procedures [3,4,5]. This is usually performed by auditing the free text reports written by the paramedics for the attended cases. EMS clinical audits need to be routinely performed to ensure adherence to treatment protocols, to identify lapses, and to discover systemic deficiencies to guide paramedic training Identification of these items for audit from the free text case reports requires a significant amount of time, resources, and effort [6].

Objectives

Methods

Results

Conclusion