A Framework for Big Earth Observation Data Using Horizontal Scaling Strategy

Abstract

The rapid growth of earth observation (EO) data poses a challenge to the way of data management. An efficient framework based on big data technology can bring new solutions. Some excellent frameworks have been proposed, which provide efficient organization and management of EO data. However, they are not optimized for data distribution in the storage environment. In this paper, an optimized EO data management strategy is proposed. Different horizontal scaling strategies are designed to explore the optimal scheme of EO data distribution. The MapReduce parallel computing model was used to test the performance of data retrieval in the experiment. The results show that the proposed strategy contributes to the efficient organization and arrangement of data. Remote sensing (RS) data blocks can be evenly distributed to different shards according to the time characteristics and hash characteristics of the strategy, and the logical index of the data reduces the time consumed by the routing process. This distributed management mode that achieves load balancing provides a framework foundation for parallel computing. Therefore, the framework with an efficient strategy can improve the performance of data management.