Commercial Cloud Providers Research Articles

Operational experience and R&D results using the Google Cloud for High-Energy Physics in the ATLAS experiment

The ATLAS experiment at CERN relies on a Worldwide Distributed Computing Grid infrastructure to support its physics program at the Large Hadron Collider. ATLAS has integrated cloud computing resources to complement its Grid infrastructure and conducted an R&D program on Google Cloud Platform. These initiatives leverage key features of commercial cloud providers: lightweight configuration and operation, elasticity and availability of diverse infrastructures. This paper examines the seamless integration of cloud computing services as a conventional Grid site within the ATLAS workflow management and data management systems, while also offering new setups for interactive, parallel analysis. It underscores pivotal results that enhance the on-site computing model and outlines several R&D projects that have benefited from large-scale, elastic resource provisioning models. Furthermore, this study discusses the impact of cloud-enabled R&D projects in three domains: accelerators and AI/ML, ARM CPUs and columnar data analysis techniques.

EFraS: Emulated framework to develop and analyze dynamic Virtual Network Embedding strategies over SDN infrastructure

The integration of Software-Defined Networking (SDN) into Network Virtualization (NV) significantly enhances network management, isolation, and troubleshooting capabilities. However, it brings forth the intricate challenge of allocating Substrate Network (SN) resources for various Virtual Network Requests (VNRs), a process known as Virtual Network Embedding (VNE). It encompasses solving two intractable sub-problems: embedding Virtual Machines (VMs) and embedding Virtual Links (VLs). While the research community has focused on formulating embedding strategies, there has been less emphasis on practical implementation at a laboratory scale, which is crucial for comprehensive design, development, testing, and validation policies for large-scale systems. However, conducting tests using commercial providers presents challenges due to the scale of the problem and associated costs. Moreover, current simulators lack accuracy in representing the complexities of communication patterns, resource allocation, and support for SDN-specific features. These limitations result in inefficient implementations and reduced adaptability, hindering seamless integration with commercial cloud providers. To address this gap, this work introduces EFraS (Emulated Framework for Dynamic VNE Strategies over SDN). The goal is to aid developers and researchers in iterating, testing, and evaluating VNE solutions seamlessly, leveraging a modular design and customized reconfigurability. EFraS offers various functionalities, including generating real-world SN topologies and VNRs. Additionally, it integrates with a diverse set of evaluation metrics to streamline the testing and validation process. EFraS leverages Mininet, Ryu controller, and OpenFlow switches to closely emulate real-time setups. Moreover, we integrate EFraS with various state-of-the-art VNE schemes, ensuring the effective validation of embedding algorithms.

IoT Serverless Computing at the Edge: A Systematic Mapping Review

Serverless computing is a new concept allowing developers to focus on the core functionality of their code, while abstracting away the underlying infrastructure. Even though there are existing commercial serverless cloud providers and open-source solutions, dealing with the explosive growth of new Internet of Things (IoT) devices requires more efficient bandwidth utilization, reduced latency, and data preprocessing closer to the source, thus reducing the overall data volume and meeting privacy regulations. Moving serverless computing to the edge of the network is a topic that is actively being researched with the aim of solving these issues. This study presents a systematic mapping review of current progress made to this effect, analyzing work published between 1 January 2015 and 1 September 2021. Using a document selection methodology which emphasizes the quality of the papers obtained through querying several popular databases with relevant search terms, we have included 64 entries, which we then further categorized into eight main categories. Results show that there is an increasing interest in this area with rapid progress being made to solve the remaining open issues, which have also been summarized in this paper. Special attention is paid to open-source efforts, as well as open-access contributions.

Transcriptome annotation in the cloud: complexity, best practices and cost

BACKGROUND The NIH Science and Technology Research Infrastructure for Discovery, Experimentation, and Sustainability (STRIDES) initiative provides NIH-funded researchers cost-effective access to commercial cloud providers, such as Amazon Web Services (AWS) and Google Cloud Platform (GCP). These cloud providers represent an alternative for the execution of large computational biology experiments like transcriptome annotation, which is a complex analytical process that requires the interrogation of multiple biological databases with several advanced computational tools. The core components of annotation pipelines published since 2012 are BLAST sequence alignments using annotated databases of both nucleotide or protein sequences almost exclusively with networked on-premises compute systems. FINDINGS We compare multiple BLAST sequence alignments using AWS and GCP. We prepared several Jupyter Notebooks with all the code required to submit computing jobs to the batch system on each cloud provider. We consider the consequence of the number of query transcripts in input files and the effect on cost and processing time. We tested compute instances with 16, 32, and 64 vCPUs on each cloud provider. Four classes of timing results were collected: the total run time, the time for transferring the BLAST databases to the instance local solid-state disk drive, the time to execute the CWL script, and the time for the creation, set-up, and release of an instance. This study aims to establish an estimate of the cost and compute time needed for the execution of multiple BLAST runs in a cloud environment. CONCLUSIONS We demonstrate that public cloud providers are a practical alternative for the execution of advanced computational biology experiments at low cost. Using our cloud recipes, the BLAST alignments required to annotate a transcriptome with ∼500,000 transcripts can be processed in <2 hours with a compute cost of ∼$200-$250. In our opinion, for BLAST-based workflows, the choice of cloud platform is not dependent on the workflow but, rather, on the specific details and requirements of the cloud provider. These choices include the accessibility for institutional use, the technical knowledge required for effective use of the platform services, and the availability of open source frameworks such as APIs to deploy the workflow.

Surviving switch failures in cloud datacenters

Switch failures can hamper access to client services, cause link congestion and blackhole network traffic. In this study, we examine the nature of switch failures in the datacenters of a large commercial cloud provider through the lens of survival theory. We study a cohort of over 180,000 switches with a variety of hardware and software configurations and find that datacenter switches have a 98% likelihood of functioning uninterrupted for over 3 months since deployment in production. However, there is significant heterogeneity in switch survival rates with respect to their hardware and software: the switches of one vendor are twice as likely to fail compared to the others. We attribute the majority of switch failures to hardware impairments and unplanned power losses. We find that the in-house switch operating system, SONiC, boosts the survival likelihood of switches in datacenters by 1% by eliminating switch failures caused by software bugs in vendor switch OSes.

Transcriptome annotation in the cloud: complexity, best practices, and cost.

The NIH Science and Technology Research Infrastructure for Discovery, Experimentation, and Sustainability (STRIDES) initiative provides NIH-funded researchers cost-effective access to commercial cloud providers, such as Amazon Web Services (AWS) and Google Cloud Platform (GCP). These cloud providers represent an alternative for the execution of large computational biology experiments like transcriptome annotation, which is a complex analytical process that requires the interrogation of multiple biological databases with several advanced computational tools. The core components of annotation pipelines published since 2012 are BLAST sequence alignments using annotated databases of both nucleotide or protein sequences almost exclusively with networked on-premises compute systems. We compare multiple BLAST sequence alignments using AWS and GCP. We prepared several Jupyter Notebooks with all the code required to submit computing jobs to the batch system on each cloud provider. We consider the consequence of the number of query transcripts in input files and the effect on cost and processing time. We tested compute instances with 16, 32, and 64 vCPUs on each cloud provider. Four classes of timing results were collected: the total run time, the time for transferring the BLAST databases to the instance local solid-state disk drive, the time to execute the CWL script, and the time for the creation, set-up, and release of an instance. This study aims to establish an estimate of the cost and compute time needed for the execution of multiple BLAST runs in a cloud environment. We demonstrate that public cloud providers are a practical alternative for the execution of advanced computational biology experiments at low cost. Using our cloud recipes, the BLAST alignments required to annotate a transcriptome with ∼500,000 transcripts can be processed in <2 hours with a compute cost of ∼$200-$250. In our opinion, for BLAST-based workflows, the choice of cloud platform is not dependent on the workflow but, rather, on the specific details and requirements of the cloud provider. These choices include the accessibility for institutional use, the technical knowledge required for effective use of the platform services, and the availability of open source frameworks such as APIs to deploy the workflow.

Cost-enabled QoS aware task scheduling in the cloud management system

Maintaining the quality of service (QoS) related parameters is an important issue in cloud management systems. The lack of such QoS parameters discourages cloud users from using the services of cloud service providers. The proposed task scheduling algorithms consider QoS parameters such as the latency, make-span, and load balancing to satisfy the user requirements. These parameters cannot sufficiently guarantee the desired user experience or that a task will be completed within a predetermined time. Therefore, this study considered the cost-enabled QoS-aware task (job) scheduling algorithm to enhance user satisfaction and maximize the profit of commercial cloud providers. The proposed scheduling algorithm estimates the cost-enabled QoS metrics of the virtual resources available from the unified resource layer in real-time. Moreover, the virtual machine (VM) manager frequently updates the current state-of-the art information about resources in the proposed scheduler to make appropriate decisions. Hence, the proposed approach guarantees profit for cloud providers in addition to providing QoS parameters such as make-span, cloud utilization, and cloud utility, as demonstrated through a comparison with existing time-and cost-based task scheduling algorithms.

Are Federated Cloud Sharing Systems Sustainable?: On Dynamic Sharing Markets and Their Stability

The recent emergence of the small cloud (SC), both in concept and in practice, has been driven mainly by issues related to service cost and complexity of commercial cloud providers (e.g., Amazon) employing massive data centers. However, the resource inelasticity problem faced by the SCs due to their relatively scarce resources might lead to a potential degradation of customer QoS and loss of revenue. A proposed solution to this problem recommends the federated sharing of resources between competing SCs to alleviate the resource inelasticity issues that might arise. Based on this idea, a recent effort proposed SC-Share, a performance-driven static market model for competitive small cloud environments that results in an efficient market equilibrium jointly optimizing customer QoS satisfaction and SC revenue generation. However, an important question with a non-obvious answer still remains to be answered, without which SC sharing markets may not be guaranteed to sustain in the long-run - is it still possible to achieve a stable market efficient state when the supply of SC resources is dynamic in nature?. In this article, we take a first step to addressing the problem of efficient market design for single SC resource sharing in dynamic environments. We answer our previous question in the affirmative through the use of Arrow and Hurwicz's disequilibrium process in economics, and the gradient play technique in game theory that allows us to iteratively converge upon efficient and stable market equilibria.

Quantitative cost comparison of on-premise and cloud infrastructure based EEG data processing

High-density, high-sampling rate EEG measurements generate large amounts of measurement data. When coupled with sophisticated processing methods, this presents a storage, computation and system management challenge for research groups and clinical units. Commercial cloud providers offer remote storage and on-demand compute infrastructure services that seem ideal for outsourcing the usually burst-like EEG processing workflow execution. There is little available guidance, however, on whether or when users should migrate to the cloud. The objective of this paper is to investigate the factors that determine the costs of on-premises and cloud execution of EEG workloads, and compare their total costs of ownership. An analytical cost model is developed that can be used for making informed decisions about the long-term costs of on-premises and cloud infrastructures. The model includes the cost-critical factors of the computing systems under evaluation, and expresses the effects of length of usage, system size, computational and storage capacity needs. Detailed cost models are created for on-premises clusters and cloud systems. Using these models, the costs of execution and data storage on clusters and in the cloud are investigated in detail, followed by a break-even analysis to determine when the use of an on-demand cloud infrastructure is preferable to on-premises clusters. The cost models presented in this paper help to characterise the cost-critical infrastructure and execution factors, and can support decision-makers in various scenarios. The analyses showed that cloud-based EEG data processing can reduce execution time considerably and is, in general, more economical when the computational and data storage requirements are relatively low. The cloud becomes competitive even in heavy load case scenarios if expensive, high quality, high-reliability clusters would be used locally. While the paper focuses on EEG processing, the models can be easily applied to CT, MRI, fMRI based neuroimaging workflows as well, which can provide guidance to the wider neuroimaging community for making infrastructure decisions.

TeraChem Cloud: A High-Performance Computing Service for Scalable Distributed GPU-Accelerated Electronic Structure Calculations.

The encapsulation and commoditization of electronic structure arise naturally as interoperability, and the use of nontraditional compute resources (e.g., new hardware accelerators, cloud computing) remains important for the computational chemistry community. We present TeraChem Cloud, a high-performance computing service (HPCS) that offers on-demand electronic structure calculations on both traditional HPC clusters and cloud-based hardware. The framework is designed using off-the-shelf web technologies and containerization to be extremely scalable and portable. Within the HPCS model, users can quickly develop new methods and algorithms in an interactive environment on their laptop while allowing TeraChem Cloud to distribute ab initio calculations across all available resources. This approach greatly increases the accessibility of hardware accelerators such as graphics processing units (GPUs) and flexibility for the development of new methods as additional electronic structure packages are integrated into the framework as alternative backends. Cost-performance analysis indicates that traditional nodes are the most cost-effective long-term solution, but commercial cloud providers offer cutting-edge hardware with competitive rates for short-term large-scale calculations. We demonstrate the power of the TeraChem Cloud framework by carrying out several showcase calculations, including the generation of 300,000 density functional theory energy and gradient evaluations on medium-sized organic molecules and reproducing 300 fs of nonadiabatic dynamics on the B800-B850 antenna complex in LH2, with the latter demonstration using over 50 Tesla V100 GPUs in a commercial cloud environment in 8 h for approximately $1250.

A Survey of Computational Tools in Solar Physics

The SunPy Project developed a 13-question survey to understand the software and hardware usage of the solar-physics community. Of the solar-physics community, 364 members across 35 countries responded to our survey. We found that 99pm 0.5% of respondents use software in their research and 66% use the Python scientific-software stack. Students are twice as likely as faculty, staff scientists, and researchers to use Python rather than Interactive Data Language (IDL). In this respect, the astrophysics and solar-physics communities differ widely: 78% of solar-physics faculty, staff scientists, and researchers in our sample uses IDL, compared with 44% of astrophysics faculty and scientists sampled by Momcheva and Tollerud (2015). 63pm 4% of respondents have not taken any computer-science courses at an undergraduate or graduate level. We also found that most respondents use consumer hardware to run software for solar-physics research. Although 82% of respondents work with data from space-based or ground-based missions, some of which (e.g. the Solar Dynamics Observatory and Daniel K. Inouye Solar Telescope) produce terabytes of data a day, 14% use a regional or national cluster, 5% use a commercial cloud provider, and 29% use exclusively a laptop or desktop. Finally, we found that 73pm 4% of respondents cite scientific software in their research, although only 42pm 3% do so routinely.

Predicting Workflow Task Execution Time in the Cloud Using A Two-Stage Machine Learning Approach

Many techniques such as scheduling and resource provisioning rely on performance prediction of workflow tasks for varying input data. However, such estimates are difficult to generate in the cloud. This paper introduces a novel two-stage machine learning approach for predicting workflow task execution times for varying input data in the cloud. In order to achieve high accuracy predictions, our approach relies on parameters reflecting runtime information and two stages of predictions. Empirical results for four real world workflow applications and several commercial cloud providers demonstrate that our approach outperforms existing prediction methods. In our experiments, our approach respectively achieves a best-case and worst-case estimation error of 1.6 and 12.2 percent, while existing methods achieved errors beyond 20 percent (for some cases even over 50 percent) in more than 75 percent of the evaluated workflow tasks. In addition, we show that the models predicted by our approach for a specific cloud can be ported with low effort to new clouds with low errors by requiring only a small number of executions.

On the Economic Sustainability of Cloud Sharing Systems Are Dynamic Single Resource Sharing Markets Stable?

The recent emergence of the small cloud (SC), both in concept and in practice, has been driven mainly by issues related to service cost and complexity of commercial cloud providers (e.g., Amazon) employing massive data centers. However, the resource inelasticity problem [29] faced by the SCs due to their relatively scarce resources might lead to a potential degradation of customer QoS and loss of revenue. A proposed solution to this problem recommends the sharing of resources between competing SCs to alleviate the resource inelasticity issues that might arise. Based on this idea, a recent e?ort ([18]) proposed SC-Share, a performance-driven static market model for competitive small cloud environments that results in an e?cient market equilibrium jointly optimizing customer QoS satisfaction and SC revenue generation. However, an important question with a non-obvious answer still remains to be answered, without which SC sharing markets may not be guaranteed to sustain in the long-run - is it still possible to achieve a stable market e?cient state when the supply of SC resources is dynamic in nature?. In this paper, we take a? rst step to addressing the problem of e?cient market design for single SC resource sharing in dynamic environments. We answer our previous question in the a?rmative through the use of Arrow and Hurwicz's disequilibrium process [9, 10] in economics, and the gradient play technique in game theory that allows us to iteratively converge upon e?cient and stable market equilibria.

Moving Just Enough Deep Sequencing Data to Get the Job Done.

Motivation:As the size of high-throughput DNA sequence datasets continues to grow, the cost of transferring and storing the datasets may prevent their processing in all but the largest data centers or commercial cloud providers. To lower this cost, it should be possible to process only a subset of the original data while still preserving the biological information of interest.Results:Using 4 high-throughput DNA sequence datasets of differing sequencing depth from 2 species as use cases, we demonstrate the effect of processing partial datasets on the number of detected RNA transcripts using an RNA-Seq workflow. We used transcript detection to decide on a cutoff point. We then physically transferred the minimal partial dataset and compared with the transfer of the full dataset, which showed a reduction of approximately 25% in the total transfer time. These results suggest that as sequencing datasets get larger, one way to speed up analysis is to simply transfer the minimal amount of data that still sufficiently detects biological signal.Availability:All results were generated using public datasets from NCBI and publicly available open source software.

Adaptive Network Coded Clouds: High Speed Downloads and Cost-Effective Version Control

Although cloud systems provide a reliable and flexible storage solution, the use of a single cloud service constitutes a single point of failure, which can compromise data availability, download speed, and security. To address these challenges, we advocate for the use of multiple cloud storage providers simultaneously using network coding as the key enabling technology. Our goal is to study two challenges of network coded storage systems. First, the efficient update of the number of coded fragments per cloud in a system aggregating multiple clouds in order to boost the download speed of files. We developed a novel scheme using recoding with limited packets to trade-off storage space, reliability, and data retrieval speed. Implementation and measurements with commercial cloud providers show that up to 9x less network use is needed compared to other network coding schemes, while maintaining similar download speeds and reliability. Second, the ability to update coded fragments from a linear erasure code when the original file is modified. We exploit code structure to provide efficient representations of the evolution of the file. Evaluations using file changes on software library repositories show that a five-order of magnitude reduction in network and storage use is possible compared to state-of-the-art.

Mastering Opportunistic Computing Resources for HEP

As results of the excellent LHC performance in 2016, more data than expected has been recorded leading to a higher demand for computing resources. It is already foreseeable that for the current and upcoming run periods a flat computing budget and the expected technology advance will not be sufficient to meet the future requirements. This results in a growing gap between supplied and demanded resources.One option to reduce the emerging lack of computing resources is the utilization of opportunistic resources such as local university clusters, public and commercial cloud providers, HPC centers and volunteer computing. However, to use opportunistic resources additional challenges have to be tackled. At the Karlsruhe Institute of Technology (KIT) an infrastructure to dynamically use opportunistic resources is built up. In this paper tools, experiences, future plans and possible improvements are discussed.

BAMSI: a multi-cloud service for scalable distributed filtering of massive genome data

BackgroundThe advent of next-generation sequencing (NGS) has made whole-genome sequencing of cohorts of individuals a reality. Primary datasets of raw or aligned reads of this sort can get very large. For scientific questions where curated called variants are not sufficient, the sheer size of the datasets makes analysis prohibitively expensive. In order to make re-analysis of such data feasible without the need to have access to a large-scale computing facility, we have developed a highly scalable, storage-agnostic framework, an associated API and an easy-to-use web user interface to execute custom filters on large genomic datasets.ResultsWe present BAMSI, a Software as-a Service (SaaS) solution for filtering of the 1000 Genomes phase 3 set of aligned reads, with the possibility of extension and customization to other sets of files. Unique to our solution is the capability of simultaneously utilizing many different mirrors of the data to increase the speed of the analysis. In particular, if the data is available in private or public clouds – an increasingly common scenario for both academic and commercial cloud providers – our framework allows for seamless deployment of filtering workers close to data. We show results indicating that such a setup improves the horizontal scalability of the system, and present a possible use case of the framework by performing an analysis of structural variation in the 1000 Genomes data set.ConclusionsBAMSI constitutes a framework for efficient filtering of large genomic data sets that is flexible in the use of compute as well as storage resources. The data resulting from the filter is assumed to be greatly reduced in size, and can easily be downloaded or routed into e.g. a Hadoop cluster for subsequent interactive analysis using Hive, Spark or similar tools. In this respect, our framework also suggests a general model for making very large datasets of high scientific value more accessible by offering the possibility for organizations to share the cost of hosting data on hot storage, without compromising the scalability of downstream analysis.

Integer linear programming-based cost optimization for scheduling scientific workflows in multi-cloud environments

Given that multi-cloud environments contain considerably diverse resources, scheduling workflows in these environments significantly reduces financial costs and overcomes the resource limitations imposed by commercial cloud providers. Accordingly, this study addressed the problem of scientific workflow scheduling in multi-cloud settings under deadline constraint to minimize associated financial costs. To this end, we proposed integer linear programming models that can be solved in a reasonable time by available solvers. In a mathematical model, the objective of a problem and real and physical constraints or restrictions are formulated using exact mathematical functions. Such formulation enabled us to comprehensively understand the system under evaluation, consider secondary preferences and post-optimality analysis and apply useful revisions to inappropriately selected input data. We analyzed the treatment of optimal cost under variations in deadline and workflow size. As part of the post-optimality analysis, sensitivity analysis and deadline revision were implemented. Results indicated that our proposed approach outperforms previously developed methods in terms of financial cost reduction.

Extending the farm on external sites: the INFN Tier-1 experience

The Tier-1 at CNAF is the main INFN computing facility offering computing and storage resources to more than 30 different scientific collaborations including the 4 experiments at the LHC. It is also foreseen a huge increase in computing needs in the following years mainly driven by the experiments at the LHC (especially starting with the run 3 from 2021) but also by other upcoming experiments such as CTA[1] While we are considering the upgrade of the infrastructure of our data center, we are also evaluating the possibility of using CPU resources available in other data centres or even leased from commercial cloud providers. Hence, at INFN Tier-1, besides participating to the EU project HNSciCloud, we have also pledged a small amount of computing resources (∼ 2000 cores) located at the Bari ReCaS[2] for the WLCG experiments for 2016 and we are testing the use of resources provided by a commercial cloud provider. While the Bari ReCaS data center is directly connected to the GARR network[3] with the obvious advantage of a low latency and high bandwidth connection, in the case of the commercial provider we rely only on the General Purpose Network. In this paper we describe the set-up phase and the first results of these installations started in the last quarter of 2015, focusing on the issues that we have had to cope with and discussing the measured results in terms of efficiency.

A world-wide databridge supported by a commercial cloud provider

Volunteer computing has the potential to provide significant additional computing capacity for the LHC experiments. One of the challenges with exploiting volunteer computing is to support a global community of volunteers that provides heterogeneous resources. However, high energy physics applications require more data input and output than the CPU intensive applications that are typically used by other volunteer computing projects. While the so-called databridge has already been successfully proposed as a method to span the untrusted and trusted domains of volunteer computing and Grid computing respective, globally transferring data between potentially poor-performing residential networks and CERN could be unreliable, leading to wasted resources usage. The expectation is that by placing a storage endpoint that is part of a wider, flexible geographical databridge deployment closer to the volunteers, the transfer success rate and the overall performance can be improved. This contribution investigates the provision of a globally distributed databridge implemented upon a commercial cloud provider.