Large-scale array analytics

Abstract

Never before in history mankind has collected data at the rates we face today. Alone in 2002, an estimated 403 Petabyte of data has been acquired, equivalent to all printed information ever created before. Earth orbiting satellites, as well as ground, airborne, and underwater sensors, space observatories scan their environment at unprecedented resolutions, giving rise to Big Science. The same holds for the life sciences where genomic data, high-resolution scans, and other modalities are collected in steadily increasing streams. Social network analysis, OLAP, and stock exchange trading represent further examples, the latter involving real-time correlation of thousands of ticker time series resulting in Terabytes of data to be analysed per single run. Summarized under Large-Scale Analytics we are witnessing an exploding demand for flexible access to massive volumes of scientific and business data sets. Arguably a large class of these massive data is represented by multi-dimensional arrays. Consequently, large arrays pose new challenges to data modelling, querying, optimization, and maintenance -- in short: we need Large-Scale Array Analytics. This tutorial introduces to the topic from a database perspective. Aspects addressed include modelling, query languages, query optimization and parallelization, and storage management. High emphasis will be devoted to applications in Big Science, particularly geo, space, and life sciences; real-life use cases will be presented and discussed which stem from our 15 years of experience with the open-source rasdaman array DBMS and our work on geo raster service standardization. We will highlight requirements, achievements, open research issues, and avenues for future research. Discussion will make use of real-life examples, many of which Internet connected participants can replay hands-on.