Spark-based in-memory DEM creation from 3D LiDAR point clouds

Abstract

ABSTRACTThe significant performance improvement obtained by using Spark in-memory processing for iterative processes has led many researchers in various fields to implement their applications with Spark. In this study, we investigated the use of in-memory processing with Spark for creating a digital elevation model from massive light detection and ranging (LiDAR) point clouds, which can be considered an iterative process. We conducted our experiments on large high-density LiDAR data sets using two well-known interpolation methods: inverse distance weighting (IDW) and Kriging. Here, we designed our in-memory processing to parallelize those methods, and compared our results with the popularly used Hadoop MapReduce-based implementation. Our experiments ran on six servers under a medium-sized high-performance cloud computing environment. The results demonstrated that our Spark-based in-memory computing yielded better performance compared with Hadoop MapReduce, with an average 5.4 times speed increase in IDW, and 4.8 times improvement in Kriging. In addition, we evaluated the characteristics of our method in terms of central processing unit, memory usage, and network activities.