Commercial Cloud Providers Research Articles

CERN Computing in Commercial Clouds

By the end of 2016 more than 10 Million core-hours of computing resources have been delivered by several commercial cloud providers to the four LHC experiments to run their production workloads, from simulation to full chain processing. In this paper we describe the experience gained at CERN in procuring and exploiting commercial cloud resources for the computing needs of the LHC experiments. The mechanisms used for provisioning, monitoring, accounting, alarming and benchmarking will be discussed, as well as the involvement of the LHC collaborations in terms of managing the workflows of the experiments within a multicloud environment.

What do We Have in Common? The Development of Common Resources in a Meta-organization

We study the development process of common resources in Helix Nebula. Helix Nebula is a meta-organization of highly heterogeneous collaborating actors, including scientific organizations like CERN, EMBL, and ESA, commercial cloud providers and public infrastructure organizations. The actors in Helix Nebula collaborate to establish a European open science cloud that can be used by scientific institutes to flexibly buy cloud computing resources from different commercial organizations. This requires the development of common resources to allow demand and supply actors to use the cloud computing infrastructure effectively. We study the process of development of common resources in Helix Nebula as an exemplary meta- organization. Our study illuminates that in the crystallizing process of common resources different development options emerge that trigger the surfacing of differences and dependencies surface. This leads to a reconfiguration of the development options, around which clusters of decision-making actors and clusters of legitimating actors shape emerging micro- alliances. Thus, our study explains the development of common resources in meta-organizations of highly heterogeneous actors without hierarchical means is an organizing process that includes a continuous reconfiguration crystallizing development options and respective micro-alliances of decision-making and legitimating actors.

Using Hadoop MapReduce for Parallel Genetic Algorithms: A Comparison of the Global, Grid and Island Models.

The need to improve the scalability of Genetic Algorithms (GAs) has motivated the research on Parallel Genetic Algorithms (PGAs), and different technologies and approaches have been used. Hadoop MapReduce represents one of the most mature technologies to develop parallel algorithms. Based on the fact that parallel algorithms introduce communication overhead, the aim of the present work is to understand if, and possibly when, the parallel GAs solutions using Hadoop MapReduce show better performance than sequential versions in terms of execution time. Moreover, we are interested in understanding which PGA model can be most effective among the global, grid, and island models. We empirically assessed the performance of these three parallel models with respect to a sequential GA on a software engineering problem, evaluating the execution time and the achieved speedup. We also analysed the behaviour of the parallel models in relation to the overhead produced by the use of Hadoop MapReduce and the GAs' computational effort, which gives a more machine-independent measure of these algorithms. We exploited three problem instances to differentiate the computation load and three cluster configurations based on 2, 4, and 8 parallel nodes. Moreover, we estimated the costs of the execution of the experimentation on a potential cloud infrastructure, based on the pricing of the major commercial cloud providers. The empirical study revealed that the use of PGA based on the island model outperforms the other parallel models and the sequential GA for all the considered instances and clusters. Using 2, 4, and 8 nodes, the island model achieves an average speedup over the three datasets of 1.8, 3.4, and 7.0 times, respectively. Hadoop MapReduce has a set of different constraints that need to be considered during the design and the implementation of parallel algorithms. The overhead of data store (i.e., HDFS) accesses, communication, and latency requires solutions that reduce data store operations. For this reason, the island model is more suitable for PGAs than the global and grid model, also in terms of costs when executed on a commercial cloud provider.

BigData and computing challenges in high energy and nuclear physics

In this contribution we discuss the various aspects of the computing resource needs experiments in High Energy and Nuclear Physics, in particular at the Large Hadron Collider. This will evolve in the future when moving from LHC to HL-LHC in ten years from now, when the already exascale levels of data we are processing could increase by a further order of magnitude. The distributed computing environment has been a great success and the inclusion of new super-computing facilities, cloud computing and volunteering computing for the future is a big challenge, which we are successfully mastering with a considerable contribution from many super-computing centres around the world, academic and commercial cloud providers. We also discuss R&D computing projects started recently in National Research Center ``Kurchatov Institute''

MC-GenomeKey: a multicloud system for the detection and annotation of genomic variants

BackgroundNext Generation Genome sequencing techniques became affordable for massive sequencing efforts devoted to clinical characterization of human diseases. However, the cost of providing cloud-based data analysis of the mounting datasets remains a concerning bottleneck for providing cost-effective clinical services. To address this computational problem, it is important to optimize the variant analysis workflow and the used analysis tools to reduce the overall computational processing time, and concomitantly reduce the processing cost. Furthermore, it is important to capitalize on the use of the recent development in the cloud computing market, which have witnessed more providers competing in terms of products and prices.ResultsIn this paper, we present a new package called MC-GenomeKey (Multi-Cloud GenomeKey) that efficiently executes the variant analysis workflow for detecting and annotating mutations using cloud resources from different commercial cloud providers. Our package supports Amazon, Google, and Azure clouds, as well as, any other cloud platform based on OpenStack. Our package allows different scenarios of execution with different levels of sophistication, up to the one where a workflow can be executed using a cluster whose nodes come from different clouds. MC-GenomeKey also supports scenarios to exploit the spot instance model of Amazon in combination with the use of other cloud platforms to provide significant cost reduction. To the best of our knowledge, this is the first solution that optimizes the execution of the workflow using computational resources from different cloud providers.ConclusionsMC-GenomeKey provides an efficient multicloud based solution to detect and annotate mutations. The package can run in different commercial cloud platforms, which enables the user to seize the best offers. The package also provides a reliable means to make use of the low-cost spot instance model of Amazon, as it provides an efficient solution to the sudden termination of spot machines as a result of a sudden price increase. The package has a web-interface and it is available for free for academic use.

SLA-based Secure Cloud Application Development

The perception of lack of control over resources deployed in the cloud may represent one of the critical factors for an organization to decide to cloudify or not its own services. The flat security features offered by commercial cloud providers to every customer, from simple practitioners to managers of huge amounts of sensitive data and services, is an additional problem. In recent years, the concept of Security Service Level Agreements (Security SLAs) is assuming a key role for the secure provisioning of cloud resources and services. This paper illustrates how to develop cloud applications that deliver services covered by Security SLAs by means of the services and tools provided by the SPECS framework, developed in the context of the SPECS (Secure Provisioning of Cloud Services based on SLA Management) European Project. The whole (SPECS) application life cycle is dealt with, in order to give a comprehensive view of the different parties involved and of the processes needed to offer security guarantees on top of cloud services. The discussed development process is exemplified by means of a real-world case study consisting in a cloud application offering a secure web container service.

ATLAS user analysis on private cloud resources at GoeGrid

User analysis job demands can exceed available computing resources, especially before major conferences. ATLAS physics results can potentially be slowed down due to the lack of resources. For these reasons, cloud research and development activities are now included in the skeleton of the ATLAS computing model, which has been extended by using resources from commercial and private cloud providers to satisfy the demands. However, most of these activities are focused on Monte-Carlo production jobs, extending the resources at Tier-2. To evaluate the suitability of the cloud-computing model for user analysis jobs, we developed a framework to launch an ATLAS user analysis cluster in a cloud infrastructure on demand and evaluated two solutions. The first solution is entirely integrated in the Grid infrastructure by using the same mechanism, which is already in use at Tier-2: A designated Panda-Queue is monitored and additional worker nodes are launched in a cloud environment and assigned to a corresponding HTCondor queue according to the demand. Thereby, the use of cloud resources is completely transparent to the user. However, using this approach, submitted user analysis jobs can still suffer from a certain delay introduced by waiting time in the queue and the deployed infrastructure lacks customizability. Therefore, our second solution offers the possibility to easily deploy a totally private, customizable analysis cluster on private cloud resources belonging to the university.

A Model Driven Framework for Portable Cloud Services: Proof of Concept Implementation

Rapid development of Cloud Computing and its increasing popularity in recent years has driven many commercial cloud providers in the market. Cloud service providers have a lot of heterogeneity in the resources they use. They have their own servers, different cloud infrastructures and APIs and methods to access the cloud resources. Lack of standards has caused the collaboration and portability of cloud services a very complex task. In this paper we have identified the challenges involved in portability of cloud apps and analyzed the existing techniques for portability at platform level. In this paper, we propose an approach using Model Driven Engineering to develop SaaS applications in a cloud-agnostic way. We introduce DSkyL, an eclipse plugin for cloud application development using feature models and domain model analysis, which would support construction, customization, development and deployment of cloud application components across multiple clouds. It also reduces the application development time drastically. This paper aims to sketch the architecture of DSkyL and the major steps involved in the process.

Network health and e-Science in commercial clouds

This paper explores the potential for improving the performance of e-Science applications on commercial clouds through the detailed examination, and characterization, of the underlying cloud network using network tomography. Commercial cloud providers are increasingly offering high performance and GPU-enabled resources that are ideal for many e-Science applications. However, the opacity of the cloud’s internal network, while a necessity for elasticity, limits the options for e-Science programmers to build efficient and high performance codes. We introduce health indicators, markers, metrics, and score as part of a network health system that provides a model for describing the overall network health of an e-Science application. We then explore the suitability of a range of tomographic techniques to act as health indicators using two testbeds—the second of which spanned one hundred AWS instances. Finally, we evaluate our work using a real-world medical image reconstruction application.

Enhanced learning classifier to locate data in cloud data centres

Cloud subscribers would like to verify the location of outsourced data in the cloud data centres to ensure that the availability of data satisfies the service level agreement SLA. Cloud users may not have access to their outsourced data in the event of operational failures in data centres or occurrence of natural disasters and/or power outages. Recently, IP geolocation techniques have been proposed to locate data files in cloud data centres. However, these techniques exploit relationships between internet delays and distance and are not extensible to incorporate different network measurements, which may be used along with internet delay to improve accuracy. Also, most of the existing techniques have only been validated with one cloud provider Amazon Web Services. In this paper, we propose an enhanced learning classier IP geolocation algorithm, which incorporates multiple network measurements to improve the accuracy of geolocating data files in data centres in four commercial cloud providers. To demonstrate the accuracy of our approach, we evaluate the performance on Amazon Web Services, Microsoft Azure, Google App Engine, and Rackspace. Our experimental results demonstrate that our approach is geolocating data files accurately, more closely to the true location and also detecting violation of location restrictions.

Framework for Platform Agnostic Enterprise Application Development Supporting Multiple Clouds

Abstract Rapid development of Cloud Computing and its increasing popularity in recent years has driven many commercial cloud providers in the market. Cloud service providers have a lot of heterogeneity in the resources they use. They have their own servers, different cloud infrastructures and APIs and methods to access the cloud resources. Lack of standards has caused the collaboration and portability of cloud services a very complex task. In this paper we have identified the challenges involved in portability of cloud apps and analyzed the existing techniques for portability at platform level. In this paper, we propose an approach using Model Driven Engineering to develop SaaS applications like CRM in a cloud-agnostic way. We introduce DSkyL, an eclipse plugin for cloud application development using feature models and domain model analysis, which would support construction, customization, development and deployment of cloud application components across multiple clouds. It also reduces the application development time drastically. This paper aims to sketch the architecture of DSkyL and the major steps involved in the development and deployment process.

Agents-based deployment of heterogeneous IaaS clouds

Cloud technology has provided much more effective and flexible management of computing infrastructure both to service vendors and customers. However, the provisioning of computing resources and deployment of applications using heterogeneous infrastructure as a service IaaS clouds is not straight forward, because offers are not easily comparable, providers ask for using different technologies and a standard is still lacking. Moreover cloud providers often hinder the adoption of a common standard in order to lock customers to their solutions. To address these issues we have designed and developed, within the research activities of the mOSAIC project, an agents-based solution that provides a uniform interface to IaaS provisioning and management. The Cloud Agency uses a set of vendor agents implementing adapters to different commercial cloud providers and open source cloud technologies. Starting from the computational constraints and from the application requirements, the vendor agents allow the users to negotiate the best IaaS solution that is available in the cloud market and to build and deploy his/her cloud infrastructure.

The EGI Federated Cloud e-Infrastructure

Abstract The EGI Federated Cloud is a standards-based open cloud system that offers a scalable and flexible e-infrastructure to the European research community, and extends the EGI computational offer beyond the traditional High Throughput Computing of the grid plat- form with new models like long-lived services and on demand computation. The EGI Cloud technology enables the federation of institutional clouds to run scientists workloads, simulations and services spanned across multiple administrative locations, allowing the users to access and exploit its resources as a unique system. The architecture of the federation was defined after a two year period of development based on a set of user stories describing operations on a cloud infrascturues and was officially launched in May 2014. Since then the EGI Federated Cloud operates as a federation of heterogeneous Infrastructure as a Service type clouds with every participating provider implementing the same set of interfaces towards users and system administrators. Enforcing cloud technology agnosticism and of supporting service mobility by means of open standards has also allowed for the inclusion of commercial cloud providers into an infrastructure previously supported only by academic institutions. This contributes to a wider goal of the funders to create economic and social impact from supported research activities. This paper details the state of the art, the design and development processes, and the organisational effort that have lead to the creation and deployment of the EGI Cloud Platform.

Cloud Bursting with GlideinWMS: Means to satisfy ever increasing computing needs for Scientific Workflows

Scientific communities have been in the forefront of adopting new technologies and methodologies in the computing. Scientific computing has influenced how science is done today, achieving breakthroughs that were impossible to achieve several decades ago. For the past decade several such communities in the Open Science Grid (OSG) and the European Grid Infrastructure (EGI) have been using GlideinWMS to run complex application workflows to effectively share computational resources over the grid. GlideinWMS is a pilot-based workload management system (WMS) that creates on demand, a dynamically sized overlay HTCondor batch system on grid resources. At present, the computational resources shared over the grid are just adequate to sustain the computing needs. We envision that the complexity of the science driven by "Big Data" will further push the need for computational resources. To fulfill their increasing demands and/or to run specialized workflows, some of the big communities like CMS are investigating the use of cloud computing as Infrastructure-As-A-Service (IAAS) with GlideinWMS as a potential alternative to fill the void. Similarly, communities with no previous access to computing resources can use GlideinWMS to setup up a batch system on the cloud infrastructure. To enable this, the architecture of GlideinWMS has been extended to enable support for interfacing GlideinWMS with different Scientific and commercial cloud providers like HLT, FutureGrid, FermiCloud and Amazon EC2. In this paper, we describe a solution for cloud bursting with GlideinWMS. The paper describes the approach, architectural changes and lessons learned while enabling support for cloud infrastructures in GlideinWMS.

ATLAS Cloud R&D

The computing model of the ATLAS experiment was designed around the concept of grid computing and, since the start of data taking, this model has proven very successful. However, new cloud computing technologies bring attractive features to improve the operations and elasticity of scientific distributed computing. ATLAS sees grid and cloud computing as complementary technologies that will coexist at different levels of resource abstraction, and two years ago created an R&D working group to investigate the different integration scenarios. The ATLAS Cloud Computing R&D has been able to demonstrate the feasibility of offloading work from grid to cloud sites and, as of today, is able to integrate transparently various cloud resources into the PanDA workload management system. The ATLAS Cloud Computing R&D is operating various PanDA queues on private and public resources and has provided several hundred thousand CPU days to the experiment. As a result, the ATLAS Cloud Computing R&D group has gained a significant insight into the cloud computing landscape and has identified points that still need to be addressed in order to fully utilize this technology.This contribution will explain the cloud integration models that are being evaluated and will discuss ATLAS' learning during the collaboration with leading commercial and academic cloud providers.

Helix Nebula and CERN: A Symbiotic approach to exploiting commercial clouds

The recent paradigm shift toward cloud computing in IT, and general interest in "Big Data" in particular, have demonstrated that the computing requirements of HEP are no longer globally unique. Indeed, the CERN IT department and LHC experiments have already made significant R&D investments in delivering and exploiting cloud computing resources. While a number of technical evaluations of interesting commercial offerings from global IT enterprises have been performed by various physics labs, further technical, security, sociological, and legal issues need to be address before their large-scale adoption by the research community can be envisaged.Helix Nebula – the Science Cloud is an initiative that explores these questions by joining the forces of three European research institutes (CERN, ESA and EMBL) with leading European commercial IT enterprises. The goals of Helix Nebula are to establish a cloud platform federating multiple commercial cloud providers, along with new business models, which can sustain the cloud marketplace for years to come.This contribution will summarize the participation of CERN in Helix Nebula. We will explain CERN's flagship use-case and the model used to integrate several cloud providers with an LHC experiment's workload management system. During the first proof of concept, this project contributed over 40.000 CPU-days of Monte Carlo production throughput to the ATLAS experiment with marginal manpower required. CERN's experience, together with that of ESA and EMBL, is providing a great insight into the cloud computing industry and highlighted several challenges that are being tackled in order to ease the export of the scientific workloads to the cloud environments.

A personalized IPTV channel-recommendation mechanism based on the MapReduce framework

Internet protocol television viewers spend considerable time browsing through the many existing channels, which is inefficient and time consuming. Although the recommendation system can solve the channel-switching problem, its performance is slow unless it is adapted to read a large amount of data sets. This study proposes a novel cloud-assisted channel-recommendation system under a cloud computing environment, channel association rules (CARs), to speed up the performance of channel switching, thereby help users to find their favorite channels in less time. The CARs algorithm is compared with the conventional (COV) solution and the most frequently selected (MFS) algorithm based on MovieLens data sets. The experimental results indicate that the predictive accuracy of CARs is superior to that of the COV and MFS algorithms. In addition, CARs use parallel computing in MapReduce to distribute large amounts of user history logs across multiple computers for processing. The experimental results show that the proposed algorithm can be employed to efficiently handle big data in a finite time when a huge of cloud servers are rented from commercial cloud providers such as Amazon Elastic Compute Cloud (EC2), Microsoft HDinsight.

Privacy-preserving computation of participatory noise maps in the cloud

This paper presents a privacy-preserving system for participatory sensing, which relies on cryptographic techniques and distributed computations in the cloud. Each individual user is represented by a personal software agent, deployed in the cloud, where it collaborates on distributed computations without loss of privacy, including with respect to the cloud service providers. We present a generic system architecture involving a cryptographic protocol based on a homomorphic encryption scheme for aggregating sensing data into maps, and demonstrate security in the Honest-But-Curious model both for the users and the cloud service providers. We validate our system in the context of NoiseTube, a participatory sensing framework for noise pollution, presenting experiments with real and artificially generated data sets, and a demo on a heterogeneous set of commercial cloud providers. To the best of our knowledge our system is the first operational privacy-preserving system for participatory sensing. While our validation pertains to the noise domain, the approach used is applicable in any crowd-sourcing application relying on location-based contributions of citizens where maps are produced by aggregating data – also beyond the domain of environmental monitoring.

SLA-based operations of massively multiplayer online games in clouds

Successful massively multiplayer online games (MMOGs) have today millions of registered users and hundreds of thousands of active concurrent players. To be able to guarantee quality of service (QoS) to a highly variable number of concurrent users, game operators statically over-provision a large infrastructure capable of sustaining the game peak load, even though a large portion of the resources is unused most of the time. To address this problem, we introduce in this work a new MMOG ecosystem for hosting and provisioning of MMOGs which effectively splits the traditional monolithic MMOG companies into three main service providers: game providers, game operators, and resource providers. Their interaction is regulated through comprehensive service level agreements (SLA) that establish the price, terms of operation, and compensation for service violations. In our model, game operators efficiently provision resources for MMOGs from multiple cloud providers, based on dynamic load forecasts, and ensure proper game operation that maintains the required QoS to all clients under varying resource availability. Game providers manage multiple distributed MMOGs for which they lease services under strict operational SLAs from game operators to satisfy all client requests. These three self-standing, smaller, more agile service providers enable access to the MMOG market for the small and medium enterprises, and to the current commercial cloud providers. We evaluate, through simulations based on real-life MMOG traces and commercial cloud SLAs, the impact of resource availability on the QoS offered to the MMOG clients. We find that our model can mitigate the negative effects of resource failures within four minutes and that MMOG server consolidation can accentuate the negative effects of failures in a resource-scarce environment. We further investigate different methods of ranking MMOG operational offers with either single or multiple (competing) MMOG providers. Our results show that compensations for SLA faults in the offer selection process can lead up to 11---16 % gain in the game providers' income. We also demonstrate that adequate ranking of offers can lead to MMOG operational cost reductions from 20 up to 60 %.

Coarse-Grain QoS-Aware Dynamic Instance Provisioning for Interactive Workload in the Cloud

Cloud computing paradigm renders the Internet service providers (ISPs) with a new approach to deliver their service with less cost. ISPs can rent virtual machines from the Infrastructure-as-a-Service (IaaS) provided by the cloud rather than purchasing them. In addition, commercial cloud providers (CPs) offer diverse VM instance rental services in various time granularities, which provide another opportunity for ISPs to reduce cost. We investigate a Coarse-grain QoS-aware Dynamic Instance Provisioning (CDIP) problem for interactive workload in the cloud from the perspective of ISPs. We formulate the CDIP problem as an optimization problem where the objective is to minimize the VM instance rental cost and the constraint is the percentile delay bound. Since the Internet traffic shows a strong self-similar property, it is hard to get an analytical form of the percentile delay constraint. To address this issue, we purpose a lookup table structure together with a learning algorithm to estimate the performance of the instance provisioning policy. This approach is further extended with two function approximations to enhance the scalability of the learning algorithm. We also present an efficient dynamic instance provisioning algorithm, which takes full advantage of the rental service diversity, to determine the instance rental policy. Extensive simulations are conducted to validate the effectiveness of the proposed algorithms.