Prototype of a production system for Cherenkov Telescope Array with DIRAC

Abstract

The Cherenkov Telescope Array (CTA) — an array of many tens of Imaging Atmospheric Cherenkov Telescopes deployed on an unprecedented scale — is the next generation instrument in the field of very high energy gamma-ray astronomy. CTA will operate as an open observatory providing data products to the scientific community. An average data stream of about 10 GB/s for about 1000 hours of observation per year, thus producing several PB/year, is expected. Large CPU time is required for data-processing as well for massive Monte Carlo simulations needed for detector calibration purposes. The current CTA computing model is based on a distributed infrastructure for the archive and the data off-line processing. In order to manage the off-line data-processing in a distributed environment, CTA has evaluated the DIRAC (Distributed Infrastructure with Remote Agent Control) system, which is a general framework for the management of tasks over distributed heterogeneous computing environments. In particular, a production system prototype has been developed, based on the two main DIRAC components, i.e. the Workload Management and Data Management Systems. After three years of successful exploitation of this prototype, for simulations and analysis, we proved that DIRAC provides suitable functionalities needed for the CTA data processing. Based on these results, the CTA development plan aims to achieve an operational production system, based on the DIRAC Workload Management System, to be ready for the start of CTA operation phase in 2017-2018. One more important challenge consists of the development of a fully automatized execution of the CTA workflows. For this purpose, we have identified a third DIRAC component, the so-called Transformation System, which offers very interesting functionalities to achieve this automatisation. The Transformation System is a ’data-driven’ system, allowing to automatically trigger data-processing and data management operations according to pre-defined scenarios. In this paper, we present a brief summary of the DIRAC evaluation done so far, as well as the future developments planned for the CTA production system. In particular, we will focus on the developments of CTA automatic workflows, based on the Transformation System. As a result, we also propose some design optimizations of the Transformation System, in order to fully support the most complex workflows, envisaged in the CTA processing.