Data Mining in Astronomy

Abstract

Like other data-rich disciplines such as physics, biology, geology, and oceanography, astronomy is facing a data avalanche due to advances in telescope and detector technology, the exponential increase in computing capabilities, improvements in data-collection methods, and successful applications of theoretical simulations. As the era approaches in which data covers the full range of wavelengths from radio to gamma-rays, the expected data volumes will add up to terabytes, soon to be followed by petabytes. Proper management and processing of massive data sets requires efficient federation of database technologies. However, mining knowledge from huge data volumes is the ultimate goal and development of data-mining techniques is therefore critical. Knowledge discovery in databases (KDD) is the process of extracting useful knowledge from data. Data mining, the application of specific algorithms to discover rare or previously unknown types of object or phenomenon, is a particular step in the process.1 KDD is inherently interactive and iterative, as shown in Figure 1. Common KDD functions are classification, cluster analysis, and regression. In classification one develops a description or model for each class of data labeled with discrete integers (as opposed to cluster analysis, which is sometimes called ‘unsupervised classification’). Classification is used for the organization of future test data, better understanding of each data class, and predictions of certain properties and behaviors. It is based on spectra or images and, for example, may be used to describe galaxies by morphology. Cluster (or clustering) analysis is a multivariate procedure based on placing objects into more or less homogeneous groups such that the relationship between groups is revealed. It lacks an underlying body of statistical theory and is heuristic in nature, requiring decisions to be made by individual users (which can strongly affect results). Cluster analysis is used to classify groups or objectsmore objectively than subjectively and can help astronomers find unusual objects within a flood of data. ExamFigure 1. Knowledge discovery in databases.