Towards Building a Distributed Data Management Architecture to Integrate Multi-Sources Remote Sensing Big Data

Abstract

Recent advances in sensor technologies have led to the significant growth of remote sensing data in terms of its volume, velocity, and variety. Remote sensing data are recognized as the big data, which pose the urgent need for new technologies to integrate and manage heterogeneous remote sensing big data. In this paper, a scalable, efficient, distributed data management system is proposed for heterogeneous remote sensing big data integration and management across the distributed environment. In the system, the remote sensing data are stored as NetCDF files. Meanwhile, the corresponding remote sensing metadata are standardized according to the ISO 19115-2:2009 based metadata template. In this way, a unified view of heterogeneous remote sensing data is formed. Two organization models are designed to store massive remote sensing data and its metadata in HDFS and HBase, respectively. Besides, an Elasticsearch-based secondary index is built for HBase to provide users with query ability of metadata contents.