The eXtreme-DataCloud project: data management services for the next generation distributed e-infrastructures

Abstract

The eXtreme-DataCloud (XDC) is an EU H2020 funded project aimed at developing scalable technologies for federating storage resources and managing data in highly distributed computing environments. XDC software stack is based on existing tools, whose technical maturity is proved, that the project enriches with new functionalities and plugins to address real life requirements from user communities belonging to a variety of scientific domains: Life Science, Astrophysics, High Energy Physics, Photon Science and Clinical Research. The XDC toolbox includes services like ONEDATA (AGH), dChace (DESY), EOS (CERN), the INDIGO-PaaS Orchestrator (INFN), DYNAFED (CERN). The targeted platforms are the current and next generation e-Infrastructures deployed in Europe, such as the European Open Science Cloud (EOSC), the European Grid Infrastructure (EGI), the Worldwide LHC Computing Grid (WLCG) and the computing infrastructures funded by other public and academic initiatives. The focus of the project is on policy driven data management, on orchestration based on Quality of Services and on pre-processing of ingested data with user-defined applications. Smart caching and storage federation technologies for the creation of the so-called “DataLakes” are also important topics for XDC. In this manuscript, we present the recent advancements in the project services architecture and the developments carried on in preparation of the first XDC release planned for the end of 2018.