Significance tests for unsupervised pattern discovery in large continuous multivariate data sets

Abstract

The question of uncertainty of discovered patterns is considered here in the context of data mining. In particular, statistical tests are developed for flagged patterns found in continuous data, where such patterns are perhaps more familiar to statisticians as local modes in the data. The significance of these patterns is indicated in terms of the probability that they have occurred by chance. The performance of these tests is examined on patterns discovered in several large data sets, including a data set describing the locations of earthquakes in California and another describing flow cytometry measurements on phytoplankton.