FTS improvements for LHC Run-3 and beyond

Edward Karavakis,Andrea Manzi,Michal Simon,Carles Garcia Cabot,Aris Angelogiannopoulos,Oliver Keeble,Mihai Patrascoiu,Maria Arsuaga Rios,C Doglioni,D Kim,W Kamleh,G.A Stewart,P Jackson,L Silvestris

doi:10.1051/epjconf/202024504016

Abstract

The File Transfer Service (FTS) developed at CERN and in production since 2014, has become a fundamental component for the LHC experiments and is tightly integrated with experiment frameworks. Starting from the beginning of 2018 with the participation to the European Commission funded project eXtreme Data Cloud (XDC) and the activities carried out in the context of the WLCG DOMA TPC and QoS working groups, a series of new developments and improvements have been planned and performed taking also into account the requirements from the experiments in preparation for the LHC Run-3. This paper provides a detailed overview of these developments; more specifically, the integration with OpenID Connect (OIDC), the QoS integration, the service scalability enhancements, the support for XRootD and HTTP Third Party Copy (TPC) transfers along with the integration with the new CERN Tape Archive (CTA) system.

Highlights

The File Transfer Service (FTS) [1, 2] is distributing the majority of the Large Hadron Collider (LHC) [3] data across the Worldwide LHC Computing Grid (WLCG) [4] infrastructure and is integrated with experiment frameworks such as Rucio [5], PhEDEx [6] and DIRAC [7]
The FTS team has been very active in performing several significant performance improvements to its core to prepare for the LHC Run-3 data challenges, supporting the new CERN Tape Archive (CTA) system, supporting a more user-friendly authentication and delegation method using tokens [20] and supporting the Third Party Copy (TPC) [21] and storage Quality of Service (QoS) [22] activities within the WLCG Data Organisation, Management and Access (DOMA) [23] project
Various performance improvements and new features were put in place in preparation for the LHC Run-3

Summary

Introduction

The File Transfer Service (FTS) [1, 2] is distributing the majority of the Large Hadron Collider (LHC) [3] data across the Worldwide LHC Computing Grid (WLCG) [4] infrastructure and is integrated with experiment frameworks such as Rucio [5], PhEDEx [6] and DIRAC [7]. FTS provides simplicity by allowing easy user interaction for submitting transfers, a WebFTS [9] portal, which is a web-based file transfer and management solution that allows users to invoke reliable, managed data transfers on distributed infrastructures from within their browser, a real-time monitoring that is rich in content and a Web Admin interface to be able to modify the internal settings of the service such as to configure access rights and limits on storages and links. It provides reliability as it ensures the data integrity since checksums are compared and failed transfers are individually retried. The FTS team has been very active in performing several significant performance improvements to its core to prepare for the LHC Run-3 data challenges, supporting the new CERN Tape Archive (CTA) system, supporting a more user-friendly authentication and delegation method using tokens [20] and supporting the Third Party Copy (TPC) [21] and storage Quality of Service (QoS) [22] activities within the WLCG Data Organisation, Management and Access (DOMA) [23] project

Performance enhancements in preparation for the LHC Run-3 data challenges

FTS support for staging with XRootD

Monitoring of migration to tape

HTTP and XRootD TPC

Work to support the WLCG DOMA QoS activity

Conclusion