Towards building a data-intensive index for big data computing – A case study of Remote Sensing data processing

Abstract

With the recent advances in Remote Sensing (RS) techniques, continuous Earth Observation is generating tremendous volume of RS data. The proliferation of RS data is revolutionizing the way in which RS data are processed and understood. Data with higher dimensionality, as well as the increasing requirement for real-time processing capabilities, have also given rise to the challenging issue of “Data-Intensive (DI) Computing”. However, how to properly identify and qualify the DI issue remains a significant problem that is worth exploring. DI computing is a complex issue. While the huge data volume may be one of the reasons for this, some other factors could also be important. In this paper, we propose an empirical model (DIRS) of DI index to estimate RS applications. DIRS here is a novel empirical model (DIRS) that could quantify the DI issues in RS data processing with a normalized DI index. Through experimental analysis of the typical algorithms across the whole RS data processing flow, we identify the key factors that affect the DI issues mostly. Finally, combined with the empirical knowledge of domain experts, we formulate DIRS model to describe the correlations between the key factors and DI index. By virtue of experimental validation on more selected RS applications, we have found that DIRS model is an easy but promising approach.