SPsync: Lightweight multi-terminal big spatiotemporal data synchronization solution

Abstract

With the rapid increase in mobile device usage and the popularity of 5G communication technology, rich Big spatiotemporal Data has become an important research resource. How to quickly synchronize the Big spatiotemporal Data collected by various types of devices to multiple distributed data centers has become the key to further utilizing spatio-temporal data. Therefore, in this paper, a new lightweight Big Spatiotemporal Data synchronization scheme called SPsync, based on a traditional file synchronization algorithm, is proposed. The contributions of SPsync are as follows. (1) SPsync leads to the optimization of the file processing strategy, changing the algorithm from a serial algorithm to a parallel one. (2) Combined with Spark, a distributed Big Data processing framework is advanced to achieve optimization of distributed tasks and further improvement of algorithm performance. Further, we demonstrate the excellent performance of SPsync through rich experiments using multiple simulated and real datasets. Compared to the best incremental synchronization algorithms currently available, the synchronization speedup is 30% faster and CPU usage 20% lower. The algorithm itself is 20% faster and CPU usage reduced by more than 25% in a multi-node scenario.