Astronomy research in big-data era

Abstract

Astronomy research has entered an increasingly data-intensive, or “big data” era. Exponential growth in the size of astronomical archives brings with it completely new requirements and challenges for data storage, computing power, networks, software, algorithms, and even research methods. Astronomers and experts in information and computation technology (ICT) are working together to simplify knowledge discovery within these massive data sets. Worldwide astronomical archives are connected through the virtual observatory (VO) framework, which provides interoperability standards and services that have allowed the creation of a global astronomical data grid. Astroinformatics, a bridge between astronomy and ICT and applied computer science, aims to engage a broader community of researchers both as contributors to and as consumers of the new methodology for data-intensive astronomy, thus building upon the data-grid foudations established by the VO framework. Data mining (DM), knowledge discovery in databases (KDD), machine learning, and visualization techniques have become a necessity. In this paper, the challenges facing modern astronomical research are briefly described, the concepts underlying the VO and astroinformatics and the latest progress in these fields are introduced, and the need for DM and KDD techniques in astronomy is discussed.