A big data pipeline for temporospatial infrasound analysis

Abstract

Smartphone and IoT sensors allow us to build dense distributed sensor networks that supplement traditional networks. We examine next-generation technologies powering the acquisition, analysis, and reporting of infrasound data. With the advent of distributed computing, managing data flow from sensor networks has become increasingly complex. Due to the increased volume, velocity, and variety of data being produced, data acquisition, storage, analysis, and reporting techniques are evolving from single server to distributed computation architectures. In collaboration with Lawrence Livermore National Laboratory, we survey and implement several Big Data technologies to tackle these issues. We implement a system that allows distributed acquisition using Akka actors, a custom time synchronization protocol, intermediate persistent queues with Apache Kafka, long term persistence using a NoSQL database, and real-time analysis and reporting with Apache Spark and Python. We describe how these software components work ...