Research on Data Sharing Architecture for Ecological Monitoring Using Iot Streaming Data

Abstract

The rapid development of Internet of Things (IoT) technology and the widespread deployment of various sensors around the world have produced a large number of data streams. Thus, current computing systems face the challenge of quickly receiving and managing these large-scale streaming data. This study builds an efficient distributed database based on Greenplum (GP) and focuses on solving the problem of the low efficiency of structured data queries for observed ecological data collected from fragile areas in Northwest China's desert oasis. First, a distributed database is designed and deployed at the physical storage structure level. A database table structure is then established based on the characteristics of the streaming data. On this basis, the data storage strategy is optimized at the data table level. Additionally, the query efficiency of the distributed database is compared with the query efficiency of traditional standalone databases. The results show that the distributed database significantly improves the data query efficiency. The greater the amount of data stored, the better the improvement in efficiency. Finally, based on the optimized distributed database, we develop a data sharing system for streaming data from ecologically fragile areas in the desert oasis in Northwest China, which provides a new approach for the efficient sharing of massive amounts of IoT streaming data for ecological monitoring. Our storage system is still currently working normally, which is highly important to both data managers and users.