Abstract
Understanding how cloud storage can be effectively used, either standalone or in support of its associated compute, is now an important consideration for WLCG.We report on a suite of extensions to familiar tools targeted at enabling the integration of cloud object stores into traditional grid infrastructures and workflows. Notable updates include support for a number of object store flavours in FTS3, Davix and gfal2, including mitigations for lack of vector reads; the extension of Dynafed to operate as a bridge between grid and cloud domains; protocol translation in FTS3; the implementation of extensions to DPM (also implemented by the dCache project) to allow 3rd party transfers over HTTP.The result is a toolkit which facilitates data movement and access between grid and cloud infrastructures, broadening the range of workflows suitable for cloud. We report on deployment scenarios and prototype experience, explaining how, for example, an Amazon S3 or Azure allocation can be exploited by grid workflows.
Highlights
With the successful exploitation of cloud CPU for simulation and the increasingly attractive price of cloud resources, attention is turning to how data-intensive workflows can be supported
While establishing a conventional grid storage system in the cloud is possible, typically through use of block devices, the overheads are considerable and, if workflows allow, it is preferable to use the provider’s own storage systems, generally an object store delivered over HTTP, such as AWS S3 [3]
The solution selected was to use the Ceph object store as a backend with Dynafed in front, providing a tactical storage system referred to as the ”data bridge”. This system offers the grid-standard interface to the experiment frameworks, which can place input data and retrieve results, and presents an alternative username/password interface to the volunteer clients, whose rights are strictly limited and who cannot use their credentials on any other storage system
Summary
- FTS3: Quantitative Monitoring H Riahi, M Salichos, O Keeble et al. - Towards an HTTP Ecosystem for HEP Data Access Fabrizio Furano, Adrien Devresse, Oliver Keeble et al. - FTS3: Quantitative Monitoring H Riahi, M Salichos, O Keeble et al. - Towards an HTTP Ecosystem for HEP Data Access Fabrizio Furano, Adrien Devresse, Oliver Keeble et al. - Resilient FTS3 service at GridKa T. Alejandro Alvarez Ayllon, Maria Arsuaga Rios, Georgios Bitzes, Fabrizio Furano, Oliver Keeble, Andrea Manzi
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
