CASOS: a subspace method for anomaly detection in high dimensional astronomical databases

Abstract

AbstractWe develop a novel algorithm for detecting anomalies. Our method has been developed to suit the challenging task of detecting anomalous sources in cross‐matched astronomical survey data. Our algorithm computes anomaly scores in lower‐dimensional subspaces of the data. By subspaces we mean, in this work, subsets of the original data variables. Our technique presents several advantages over existing methods: it can work directly on data with missing values; it addresses some of the problems posed by high‐dimensional data spaces; it is less susceptible to a masking effect from irrelevant features; it can be easily adapted to suit specific needs and it allows an easier interpretation of why a given object has a high combined anomaly score. One drawback of our method is that it cannot detect outliers that are only apparent in high‐dimensional spaces. Anomaly scores are computed using a nearest neighbor (NN) technique, but the algorithm works with any other method computing numerical anomaly scores. We demonstrate the properties of our algorithm and evaluate its performance on both simulated and real datasets. We show that it is capable of outperforming state‐of‐the‐art, full‐dimensional approaches in some situations. © 2013 Wiley Periodicals, Inc. Statistical Analysis and Data Mining 6: 53–72, 2013