A framework to monitor activities of satellite data processing in real-time

M D Nguyen,A P Kryukov

doi:10.1088/1742-6596/955/1/012037

Abstract

Space Monitoring Data Center (SMDC) of SINP MSU is one of the several centers in the world that collects data on the radiational conditions in near-Earth orbit from various Russian (Lomonosov, Electro-L1, Electro-L2, Meteor-M1, Meteor-M2, etc.) and foreign (GOES 13, GOES 15, ACE, SDO, etc.) satellites. The primary purposes of SMDC are: aggregating heterogeneous data from different sources; providing a unified interface for data retrieval, visualization, analysis, as well as development and testing new space weather models; and controlling the correctness and completeness of data. Space weather models rely on data provided by SMDC to produce forecasts. Therefore, monitoring the whole data processing cycle is crucial for further success in the modeling of physical processes in near-Earth orbit based on the collected data. To solve the problem described above, we have developed a framework called Live Monitor at SMDC. Live Monitor allows watching all stages and program components involved in each data processing cycle. All activities of each stage are logged by Live Monitor and shown in real-time on a web interface. When an error occurs, a notification message will be sent to satellite operators via email and the Telegram messenger service so that they could take measures in time. The Live Monitor’s API can be used to create a customized monitoring service with minimum coding.

Highlights

One of the most critical tasks of a space monitoring data centre is providing correct data collected from satellites which are suitable to be used in research
SDDS automates the whole cycle of satellite data processing which consists of the following steps: 1) connecting to data sources of each satellite; 2) checking for new data; 3) downloading them to a temporary storage; 4) decoding data if they were encoded; 5) extracting raw instrumental data from decoded data; 6) producing scientific data suitable for further analysis from the raw ones; 7) inserting scientific data into a unified database; 8) moving both raw and scientific data to long-term data storage with compression on demand
If an error occurs during a processing cycle, the mechanism must detect the component that caused the error and inform satellite operators and developers immediately so that they could take measure in time

Summary

Introduction

One of the most critical tasks of a space monitoring data centre is providing correct data collected from satellites which are suitable to be used in research. To be sure of the data correctness, we need a mechanism that can monitor all activities of each program during a processing cycle and produce a report at the end. The idea of the active way, so-called lightweight event-driven or push notification, is that whenever an event arises the target program sends a short message (or a signal) to a master server, and the master server, in turn, broadcasts the message to all subscribed users.

Results

Conclusion