Abstract
CloudVeneto.it was initially funded and deployed by INFN in 2014 for serving the computational and storage demands of INFN research projects mainly related to HEP and Nuclear Physics. It is an OpenStack-based scientific cloud with resources spread across two different sites connected with a high speed optical link: INFN Padova Unit and the INFN Legnaro National Laboratories. The infrastructure has grown throughout the years with additional funds from ten University of Padova departments, and nowadays supports a broader range of scientific and engineering disciplines. Its hardware resources provide around 2500 computational cores and 360 TB of storage to about 250 users working for more than 70 projects. In the last months we enhanced the cloud platform in two ways: 1) by integrating a number of heterogeneous GPU cards to address the special needs of user communities whose computations involve machine learning training; 2) by enabling the users to simply deploy on-demand Kubernetes clusters for Big Data Analytics applications taking advantage of the operator framework. In particular, the Kubernetes operators for Apache Kafka and Spark platforms were integrated to address real-time data ingestion and streaming processing on the cloud. This article describes the technical details of these two solutions and their integration with the cloud infrastructure.
Highlights
The origin and the details of the CloudVeneto.it infrastructure have been described in a previous article [1]
CloudVeneto.it was initially funded and deployed by INFN in 2014 for serving the computational and storage demands of INFN research projects mainly related to HEP and Nuclear Physics
In the last months we enhanced the cloud platform in two ways: 1) by integrating a number of heterogeneous GPU cards to address the special needs of user communities whose computations involve machine learning training; 2) by enabling the users to deploy on-demand Kubernetes clusters for Big Data Analytics applications taking advantage of the operator framework
Summary
The origin and the details of the CloudVeneto.it infrastructure have been described in a previous article [1]. CloudVeneto.it is an OpenStack based IaaS that has been funded by INFN and ten departments of the University of Padova, and is serving the scientific user communities affiliated to them. The main OpenStack services (Horizon, Keystone, Neutron, Glance, Nova, Cinder, Heat, EC2 API) are hosted in two controller nodes implementing high availability in an active-active configuration. A complex network configuration (described in detail in [1]) with four virtual routers and one or more class-C virtual networks for each OpenStack project implements the different access policies defined for INFN, University and non academic (e.g. related to collaboration projects with Public Administration or industry) users, based on the ownership of the resources. Besides the elastic on-demand HTCondor batch cluster service already provided to CloudVeneto.it users, another PaaS-type service designed to deploy a Big Data Analytics platform was developed and put in production during 2019
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
