Fast Subset Scan Research Articles

Support vector subset scan for spatial pattern detection

Discovery of localized and irregularly shaped anomalous patterns in spatial data provides useful context for operational decisions across many policy domains. The support vector subset scan (SVSS) integrates the penalized fast subset scan with a kernel support vector machine classifier to accurately detect spatial clusters without imposing hard constraints on the shape or size of the pattern. The method iterates between (1) efficiently maximizing a penalized log-likelihood ratio over subsets of locations to obtain an anomalous pattern, and (2) learning a high-dimensional decision boundary between locations included in and excluded from the anomalous subset. On each iteration, location-specific penalties to the log-likelihood ratio are assigned according to distance to the decision boundary, encouraging patterns which are spatially compact but potentially highly irregular in shape. SVSS outperforms competing methods for spatial cluster detection at the task of detecting randomly generated patterns in simulated experiments. SVSS enables discovery of practically-useful anomalous patterns for disease surveillance in Chicago, IL, crime hotspot detection in Portland, OR, and pothole cluster detection in Pittsburgh, PA, as demonstrated by experiments using publicly available data sets from these domains.

Support Vector Subset Scan for Spatial Outbreak Detection

ObjectiveWe present the support vector subset scan (SVSS), a new methodfor detecting localized and irregularly shaped patterns in spatial data.SVSS integrates the penalized fast subset scan3with a kernel supportvector machine classifier to accurately detect disease clusters that arecompact and irregular in shape.IntroductionNeill’s fast subset scan2detects significant spatial patterns ofdisease by efficiently maximizing a log-likelihood ratio statisticover subsets of locations, but may result in patterns that are notspatially compact. The penalized fast subset scan (PFSS)3providesa flexible framework for adding soft constraints to the fast subsetscan, rewarding or penalizing inclusion of individual points into acluster with additive point-specific penalty terms. We propose thesupport vector subset scan (SVSS), a novel method that iterativelyassigns penalties according to distance from the separating hyperplanelearned by a kernel support vector machine (SVM). SVSS efficientlydetects disease clusters that are geometrically compact and irregular.MethodsSpeakman3observes that for a fixed value of relative riskq, thelog-likelihood ratio for the exponential family of expectation-basedscan statistics can be written as an additive set function over all dataelements. This property enables addition of element-specific penaltyterms to the log-likelihood ratio, interpreted as the prior log-odds ofincluding a data point in the cluster. We propose an iterative methodfor setting the penalty terms which leads to spatially compact clusters,alternately running PFSS to obtain an optimal subset and traininga kernel SVM to maximize the margin between points within andoutside of the subset. On each iteration of PFSS, penalties are assignedbased on distance to the SVM decision boundary. We apply randomrestarts across the penalty space to approach a global optimum in thenon-convex SVSS objective function.ResultsWe demonstrate detection of disease clusters in mosquito poolstested for West Nile Virus (WNV), using data made publicly availableby the Chicago Department of Public Health through the City ofChicago Data Portal. In comparison to the circular scan1, whichdetects circular patterns with elevated WNV, SVSS has improvedpower to detect disease clusters that are elongated or irregularin shape. For example, the top WNV cluster detected by SVSSroughly conforms to sections of two major rivers in North Chicago,overlapping significant portions of the forest preserves adjacent tothese rivers. The unconstrained fast subset scan2has high detectionpower for subtle and irregular disease clusters, but finds patterns thatare spatially sparse and intermingled with non-anomalous points.SVSS rewards patterns with spatial coherence, detecting clustersthat are compact and separated from non-anomalous points whilemaintaining power to detect slight but significant increases in detectedrates of WNV.ConclusionsSVSS introduces soft spatial constraints to the fast subset scan2in the form of penalties to the log-likelihood ratio statistic, learnediteratively based on distance to a high-dimensional SVM decisionboundary. These constraints give SVSS greater power to detectspatially compact and irregular patterns of disease.Clusters of West Nile Virus detected by three scanning algorithms.

Penalized Fast Subset Scanning

We present the penalized fast subset scan (PFSS), a new and general framework for scalable and accurate pattern detection. PFSS enables exact and efficient identification of the most anomalous subsets of the data, as measured by a likelihood ratio scan statistic. However, PFSS also allows incorporation of prior information about each data element’s probability of inclusion, which was not previously possible within the subset scan framework. PFSS builds on two main results: first, we prove that a large class of likelihood ratio statistics satisfy a property that allows additional, element-specific penalty terms to be included while maintaining efficient computation. Second, we prove that the penalized statistic can be maximized exactly by evaluating only O(N) subsets. As a concrete example of the PFSS framework, we incorporate “soft” constraints on spatial proximity into the spatial event detection task, enabling more accurate detection of irregularly shaped spatial clusters of varying sparsity. To do so, we develop a distance-based penalty function that rewards spatial compactness and penalizes spatially dispersed clusters. This approach was evaluated on the task of detecting simulated anthrax bio-attacks, using real-world Emergency Department data from a major U.S. city. PFSS demonstrated increased detection power and spatial accuracy as compared to competing methods while maintaining efficient computation.

A false discovery approach for scanning spatial disease clusters with arbitrary shapes

ABSTRACTThe spatial scan statistic is one of the main tools for testing the presence of clusters in a geographical region. The recently proposed Fast Subset Scan (FSS) method represents an important extension, as it is computationally efficient and enables detection of clusters with arbitrary shapes. Aimed at automatically and simultaneously detecting multiple clusters of any shapes, this article explores the False Discovery (FD) approach originated from multiple hypothesis testing. We show that the FD approach can provide a higher detection power and better identification capability than the standard scan and FSS methods, on average.

StarScan: A Novel Scan Statistic for Irregularly-Shaped Spatial Clusters

We propose StarScan, a new star-shaped scan statistic for detecting irregularly-shaped spatial clusters. StarScan generalizes the traditional, circular spatial scan statistic by allowing the radius of the cluster around a center location to vary continuously with the angle, but penalizes the log-likelihood ratio score proportional to the total change in radius. StarScan was compared with circular scan and fast subset scan on simulated respiratory outbreaks and bioterrorist anthrax attacks injected into real-world Emergency Department data. Given a small amount of labeled training data, StarScan learns appropriate penalties for both compact and elongated clusters, resulting in improved detection performance.

Fast subset scan for multivariate event detection

We present new subset scan methods for multivariate event detection in massive space-time datasets. We extend the recently proposed 'fast subset scan' framework from univariate to multivariate data, enabling computationally efficient detection of irregular space-time clusters even when the numbers of spatial locations and data streams are large. For two variants of the multivariate subset scan, we demonstrate that the scan statistic can be efficiently optimized over proximity-constrained subsets of locations and over all subsets of the monitored data streams, enabling timely detection of emerging events and accurate characterization of the affected locations and streams. Using our new fast search algorithms, we perform an empirical comparison of the Subset Aggregation and Kulldorff multivariate subset scans on synthetic data and real-world disease surveillance tasks, demonstrating tradeoffs between the detection and characterization performance of the two methods.

Fast Subset Scan for Spatial Pattern Detection

SummaryWe propose a new ‘fast subset scan’ approach for accurate and computationally efficient event detection in massive data sets. We treat event detection as a search over subsets of data records, finding the subset which maximizes some score function. We prove that many commonly used functions (e.g. Kulldorff’s spatial scan statistic and extensions) satisfy the ‘linear time subset scanning’ property, enabling exact and efficient optimization over subsets. In the spatial setting, we demonstrate that proximity-constrained subset scans substantially improve the timeliness and accuracy of event detection, detecting emerging outbreaks of disease 2 days faster than existing methods.

New Directions in Artificial Intelligence for Public Health Surveillance

The next decade of disease surveillance research will require novel methods to effectively use massive quantities of complex, high-dimensional data. We summarize two recent approaches which deal with the increasing complexity and scale of health data, including the use of rich text data to detect emerging outbreaks with novel symptom patterns, and fast subset scan methods to efficiently identify the most relevant patterns in massive datasets.