Adaptive data reduction workflows for astronomy: The ESO Data Processing System (EDPS)

Abstract

Context. Astronomical data reduction is usually done via processing pipelines that consist of a series of individual processing steps that can be executed one at a time. These processing steps are then strung together into workflows and fed with data to address a particular processing goal. Examples of such pipeline processing goals are the quality control of incoming data from telescopes, unsupervised production of science and calibration products for an archive, and supervised data reduction to serve the specific science goals of a scientist. For each of these goals, individual workflows need to be developed. These workflows need to evolve when the pipeline, observing strategies, or calibration plans change. Writing and maintaining such a collection of workflows is therefore a complex and expensive task. Aims. In this paper we propose a data processing system that automatically derives processing workflows for different use cases from a single specification of a cascade of processing steps. Methods. The system works by using formalised descriptions of data processing pipelines that specify the input and output of each processing step. Inputs can be existing data or the output of a previous step. The rules for selecting the most appropriate input data are directly attached to the description. Results. A version of the proposed system has been implemented as the ESO Data Processing System (EDPS) in the Python language. The specification of processing cascades and data organisation rules use a restrictive set of Python classes, attributes, and functions. Conclusions. The EDPS implementation of the proposed system was used to demonstrate that it is possible to automatically derive from a single specification of a pipeline processing cascade the workflows that the European Southern Observatory uses for quality control, archive production, and specialised science reduction. The EDPS will be used to replace all data reduction systems using different workflow specifications that are currently in use at the European Southern Observatory.