A robust framework for real-time distributed processing of satellite data

Abstract

It is estimated that future satellite instruments such as the Advanced Baseline Imager (ABI) and the Hyperspectral Environmental Suite (HES) on the GOES-R series of satellites will provide raw data volume of about 1.5 Terabyte per day. Due to the high data rate, satellite ground data processing will require considerable computing power to process data in real-time. Cluster technologies employing a multi-processor system present the only current economically viable option. To sustain high levels of system reliability and operability in a cluster-oriented operational environment, a fault-tolerant data processing framework is proposed to provide a platform for encapsulating science algorithms for satellite data processing. The science algorithms together with the framework are hosted on a Linux cluster. In this paper we present an architectural model and a system prototype for providing performance, reliability, and scalability of candidate hardware and software for a satellite data processing system. Furthermore, benchmarking results are presented for a selected number of science algorithms for the Geosynchronous Imaging Fourier Transform Spectrometer (GIFTS) instrument showing that considerable performance can be gained without sacrificing the reliability and high availability constraints imposed on the operational cluster system.