Testing the limits of HTTPS single point third party copy transfer over the WAN

Edgar Fajardo,Frank Wuerthwein,Brian Bockelman,C Doglioni,W Kamleh,G.A Stewart,L Silvestris,D Kim,P Jackson

doi:10.1051/epjconf/202024504025

Edgar Fajardo, Frank Wuerthwein + Show 7 more

Open Access

https://doi.org/10.1051/epjconf/202024504025

Copy DOI

Abstract

LHC data is constantly being moved between computing and storage sites to support analysis, processing, and simulation; this is done at a scale that is currently unique within the science community. For example, the CMS experiment on the LHC manages approximately 200PB of storage across 100 sites and, on a daily basis, moves 1PB between sites via GridFTP as primary protocol. This paper describes the performance results we have achieved by exploring alternatives to the GridFTP protocol for these data movements. In particular the HTTPS third party copy over Xrootd data servers as a possible replacement of GridFTP for LHC big data movements.

Highlights

The current model for wide area network access of a site’s compute storage on the Worldwide LHC Computing Grid (WLCG)[1] is the storage element (SE)
In order for us to benchmark the performance of file transfers over HTTP using XRootD Third Party Copy mechanisms we set up two 100Gbit/sec network capable servers and placed them such that transfers cross several regional networks
Present, based on GridFTP servers aggregated via Linux Virtual Server (LVS), and proposed future, based on HTTP implemented via XRootD servers, architecture for WLCG file transfers are discussed

Summary

Introduction

The current model for wide area network access of a site’s compute storage on the Worldwide LHC Computing Grid (WLCG)[1] is the storage element (SE). Each tuple consist of a Storage Element (and type Xrootd, HTTP, GridFTP, etc) and a file name. In preparations for the High Luminosity Large Hadron Collider (HL-LHC) a joint taskforce between OSG and IrisHEP was setup to explore the possibility of having all the transfer done using less proprietary protocols (GridFTP) in favor of more widely used protocols: HTTP, while leveraging other existing pieces of the grid software stack: XRootD [8]. The proposed replacement for the LVS with GridFTP configuration consists of using XRootD servers behind a redirector using HTTP as transfer protocol.

Current Grid FTP LVS Architecture

XrootD Third Party Copy

Stress Tests and Current Deployment

Production Deployments of HTTP via XRootD at US CMS Tier 2 Centers

Conclusions