Abstract
The New York City Department of Health and Mental Hygiene has operated an emergency department syndromic surveillance system since 2001, using temporal and spatial scan statistics run on a daily basis for cluster detection. Since the system was originally implemented, a number of new methods have been proposed for use in cluster detection. We evaluated six temporal and four spatial/spatio-temporal detection methods using syndromic surveillance data spiked with simulated injections. The algorithms were compared on several metrics, including sensitivity, specificity, positive predictive value, coherence, and timeliness. We also evaluated each method’s implementation, programming time, run time, and the ease of use. Among the temporal methods, at a set specificity of 95%, a Holt-Winters exponential smoother performed the best, detecting 19% of the simulated injects across all shapes and sizes, followed by an autoregressive moving average model (16%), a generalized linear model (15%), a modified version of the Early Aberration Reporting System’s C2 algorithm (13%), a temporal scan statistic (11%), and a cumulative sum control chart (<2%). Of the spatial/spatio-temporal methods we tested, a spatial scan statistic detected 3% of all injects, a Bayes regression found 2%, and a generalized linear mixed model and a space-time permutation scan statistic detected none at a specificity of 95%. Positive predictive value was low (<7%) for all methods. Overall, the detection methods we tested did not perform well in identifying the temporal and spatial clusters of cases in the inject dataset. The spatial scan statistic, our current method for spatial cluster detection, performed slightly better than the other tested methods across different inject magnitudes and types. Furthermore, we found the scan statistics, as applied in the SaTScan software package, to be the easiest to program and implement for daily data analysis.
Highlights
In the autumn of 2001 the New York City (NYC) Department of Health and Mental Hygiene (DOHMH) established an emergency department (ED) syndromic surveillance system to detect and track disease outbreaks and illness in the city
We did not submit this project to the NYC DOHMH Institutional Review Board, as the existing data are collected as part of routine public health surveillance, could not be linked to individuals, and were analyzed anonymously
For our final temporal method, we evaluated the SaTScan temporal scan statistic using a discrete Bernoulli probability model, which is used by our current system, and has been previously described [2]
Summary
While the majority of these studies have found good sensitivity with detecting large, rapid increases in cases, most have been less successful with finding smaller outbreaks, which are more likely to occur in practice These studies show that using as much historical data that is available can be beneficial [20], as well as testing methods on multiple outbreak types and data streams of differing baseline counts [15,20], including varying day of week and seasonal patterns [21]. Adjusting for these temporal patterns can significantly improve performance [14,20,23]. Few of these studies compared performance between multiple spatial/spatio-temporal methods
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
