A Distributed Data Processing Platform over Meteorological Big Data Using MapReduce

Abstract

In the era of big data, the data of the meteorological departments grows explosively, which puts higher requirements on the real-time processing of meteorological big data. Besides, the efficient storage for the massive meteorological data has also attracted much attention from the meteorological departments. Therefore, in response to the urgent requirements of meteorological big data in processing and storage, a distributed data processing platform over meteorological big data using MapReduce is designed. Technically, the platform develops corresponding real-time strategies according to various data properties, and obtains meteorological big data in real-time from multiple channels. Based on the MapReduce real-time processing, we realise the distributed storage to store the meteorological big data in the platform. Overall, our platform improves the real-time, reliability, availability and the access efficiency of meteorological big data which is easy to expand and also has a good reference value for big data processing in other similar industries.