Forward Observer system for radar data workflows: Big data management in the field

Abstract

There are unique challenges in managing data collection and management from instruments in the field in general. These issues become extreme when “in the field” means “in a plane over the Antarctic”. In this paper we present the design and function of the Forward Observer a computer cluster and data analysis system that flies in a plane in the Arctic and Antarctic to collect, analyze in real time, and store Synthetic Aperture Radar (SAR) data. SAR is used to analyze the thickness and structure of polar ice sheets. We also discuss the processing of data once it is returned to the continental US and made available via data grids. The needs for in-flight data analysis and storage in the Antarctic and Arctic are highly unusual, and we have developed a novel system to meet those needs.We describe the constraints and requirements that led to the creation of this system and the general functionality which it applies to any instrument. We discuss the main means for handling replication and creating checksum information to ensure that data collected in polar regions are returned safely to mainland US for analysis. So far, not a single byte of data collected in the field has failed to make it home to the US for analysis (although many particular data storage devices have failed or been damaged due to the challenges of the extreme environments in which this system is used).While the Forward Observer system is developed for the extreme situation of data management in the field in the Antarctic, the technology and solutions we have developed are applicable and potentially usable in many situations where researchers wish to do real time data management in the field in areas that are constrained in terms of electrical supply.