A study of the transferability of influenza case detection systems between two large healthcare systems.

Ye Ye,Howard Su,Per H Gesteland,Lingyun Shi,Michael M Wagner,John M Aronis,Thomas Ginter,Nicholas E Millett,Fuchiang Tsui,Victor M Ruiz,Gregory F Cooper,Andrew J Nowalk,Rudy Van Bree,Peter J Haug,Jeffrey P Ferraro,Arturo López Pineda,Jen-Hsiang Chuang

doi:10.1371/journal.pone.0174970

Abstract

ObjectivesThis study evaluates the accuracy and transferability of Bayesian case detection systems (BCD) that use clinical notes from emergency department (ED) to detect influenza cases.MethodsA BCD uses natural language processing (NLP) to infer the presence or absence of clinical findings from ED notes, which are fed into a Bayesain network classifier (BN) to infer patients’ diagnoses. We developed BCDs at the University of Pittsburgh Medical Center (BCDUPMC) and Intermountain Healthcare in Utah (BCDIH). At each site, we manually built a rule-based NLP and trained a Bayesain network classifier from over 40,000 ED encounters between Jan. 2008 and May. 2010 using feature selection, machine learning, and expert debiasing approach. Transferability of a BCD in this study may be impacted by seven factors: development (source) institution, development parser, application (target) institution, application parser, NLP transfer, BN transfer, and classification task. We employed an ANOVA analysis to study their impacts on BCD performance.ResultsBoth BCDs discriminated well between influenza and non-influenza on local test cases (AUCs > 0.92). When tested for transferability using the other institution’s cases, BCDUPMC discriminations declined minimally (AUC decreased from 0.95 to 0.94, p<0.01), and BCDIH discriminations declined more (from 0.93 to 0.87, p<0.0001). We attributed the BCDIH decline to the lower recall of the IH parser on UPMC notes. The ANOVA analysis showed five significant factors: development parser, application institution, application parser, BN transfer, and classification task.ConclusionWe demonstrated high influenza case detection performance in two large healthcare systems in two geographically separated regions, providing evidentiary support for the use of automated case detection from routinely collected electronic clinical notes in national influenza surveillance. The transferability could be improved by training Bayesian network classifier locally and increasing the accuracy of the NLP parser.

Highlights

The control of epidemic diseases is an increasingly important problem whose solution rests, in part, on improvements in the methods for disease surveillance
A Bayesian case detection systems (BCD) uses natural language processing (NLP) to infer the presence or absence of clinical findings from emergency department (ED) notes, which are fed into a Bayesain network classifier (BN) to infer patients’ diagnoses
This study demonstrated that a case detection system developed in the University of Pittsburgh Medical Center (i.e., BCDUPMC) could be deployed without modification at the Intermountain Healthcare for use in public health influenza surveillance, despite being in different regions of the country

Summary

Introduction

The control of epidemic diseases is an increasingly important problem whose solution rests, in part, on improvements in the methods for disease surveillance. Elkin et al [9] found that a regression model for influenza case detection using whole encounter notes was more accurate than a model that uses only the chief complaint field in the encounter notes (area under the receiver operating characteristic curve (AUC): 0.764 vs 0.652). They concluded that the national strategy for biosurveillance should be changed from chief complaints to encounter notes

Objectives

Methods

Results

Discussion

Conclusion