Implementation of a distributed processing engine for spatial big-data processing based on batch and stream

Abstract

Recently, the amount of spatial information generated in real time due to the use of SNS is increasing rapidly. Research is also actively underway to extract only the information of interest from real-time spatial information. Then, the spatial information generated in the SNS can be requested by the API of the corresponding SNS company. In this case, the responding spatial information is provided to the requester in the form of stream in JSON format, and a dedicated ETL tool capable of processing the spatial information of the stream type is needed. In order to continuously process the massive spatial information of the stream type, the distributed and parallel processing method should be used rather than the existing single processing method. At this time, the necessary ETL tools are distributed and parallel ETL tools. In this paper, we design and implement a spatial ETL processing engine that can process spatial information on a stream basis as well as an ETL processing spatial information on the existing layout base. The spatial ETL processing engine proposed in this paper is a spatial ETL engine for distributed processing. For the general user, the unit module experiment was commissioned by TTA(Telecommunication Technology Association).