Abstract
The shear volumes of data generated from earth observation and remote sensing technologies continue to make major impact; leaping key geospatial applications into the dual data and compute-intensive era. As a consequence, this rapid advancement poses new computational and data processing challenges. We implement a novel remote sensing data flow (RESFlow) for advancing machine learning to compute with massive amounts of remotely sensed imagery. The core contribution is partitioning massive amounts of data into homogeneous distributions for fitting simple models. RESFlow takes advantage of Apache Spark and the availability of modern computing hardware to harness the acceleration of deep learning inference on expansive remote sensing imagery. The framework incorporates a strategy to optimize resource utilization across multiple executors assigned to a single worker. We showcase its deployment in both computationally and data-intensive workloads for pixel-level labeling tasks. The pipeline invokes deep learning inference at three stages; during deep feature extraction, deep metric mapping, and deep semantic segmentation. The tasks impose compute-intensive and GPU resource sharing challenges motivating for a parallelized pipeline for all execution steps. To address the problem of hardware resource contention, our containerized workflow further incorporates a novel GPU checkout routine and the ticketing system across multiple workers. The workflow is demonstrated with NVIDIA DGX accelerated platforms and offers appreciable compute speed-ups for deep learning inference on pixel labeling workloads; processing 21 028 TB of imagery data and delivering output maps at area rate of 5.245 sq.km/s, amounting to 453 168 sq.km/day—reducing a 28 day workload to 21 h.
Highlights
E ARTH observation and remote-sensing are both fields that have undergone a renaissance recently, making major impacts in key geospatial applications including land cover mapping, infrastructure mapping, damage assessment, and population distribution studies [1]–[4]
2) We take advantage of Apache Spark to provide, for a single large image scene, fast parallel inference functionality wherein an area pixel labeling rate of 5.245 sq.km/s, amounting to 453 168 sq.km/day is achieved—reducing a 28 day workload to 21 h. 3) We present a containerized workflow for Apache Spark operations coordinated with GPUs for deep learning inference best practices, e.g., efficient GPU usage and ticketing across multiple workers, for large deep learning workloads deployed on GPU clusters
remote sensing data flow (RESFlow) is seen to perform very to the Mono model for two of the three test regions. This is considerable, as each model from the RESFlow Image Gallery sees considerably less data compared to its Mono model counterpart during training, and yet is able to generalize to a similar degree
Summary
E ARTH observation and remote-sensing are both fields that have undergone a renaissance recently, making major impacts in key geospatial applications including land cover mapping, infrastructure mapping, damage assessment, and population distribution studies [1]–[4]. Remote sensing applications have leaped into a data and compute-intensive era presenting challenges and opportunities for new advanced machine learning and computer vision workflows. Examples of such applications include supporting accurate population distribution estimates, possibilities to study sustainability outcomes at scale [5], and identifying urban environments over large contexts using abundant satellite imagery and breakthroughs in deep learning based image classification [6]
Sign-in/Register to view highlights & summary 
Published Version (
Free)
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call