Cloud Instances Research Articles

PLI $$^+$$ + : efficient clustering of cloud databases

Commercial cloud database services increase availability of data and provide reliable access to data. Routine database maintenance tasks such as clustering, however, increase the costs of hosting data on commercial cloud instances. Clustering causes an I/O burst; clustering in one-shot depletes I/O credit accumulated by an instance and increases the cost of hosting data. An unclustered database decreases query performance by scanning large amounts of data, gradually depleting I/O credits. In this paper, we introduce Physical Location Index Plus ( $${PLI}^{\small {{+}}}$$ ), an indexing method for databases hosted on commercial cloud. $${PLI}^{\small {{+}}}$$ relies on internal knowledge of data layout, building a physical location index, which maps a range of physical co-locations with a range of attribute values to create approximately sorted buckets. As new data is inserted, writes are partitioned in memory based on incoming data distribution. The data is written to physical locations on disk in block-based partitions to favor large granularity I/O. Incoming SQL queries on indexed attribute values are rewritten in terms of the physical location ranges. As a result, $${PLI}^{\small {{+}}}$$ does not decrease query performance on an unclustered cloud database instance, DBAs may choose to cluster the instance when they have sufficiently large I/O credit available for clustering thus delaying the need for clustering. We evaluate query performance over $${PLI}^{\small {{+}}}$$ by comparing it with clustered, unclustered (secondary) indexes, and log-structured merge trees on real datasets. Experiments show that $${PLI}^{\small {{+}}}$$ significantly delays clustering, and yet does not degrade query performance—thus achieving higher level of sortedness than unclustered indexes and log-structured merge trees. We also evaluate the quality of clustering by introducing a measure of interval sortedness, and the size of index.

Truthful Online Auction Toward Maximized Instance Utilization in the Cloud

Although infrastructure as a service (IaaS) users are busy scaling up/out their cloud instances to meet the ever-increasing demands, the dynamics of their demands, as well as the coarse-grained billing options offered by leading cloud providers, have led to substantial instance underutilization in both temporal and spatial domains. This paper systematically examines an instance subletting service, where sublettable instances can be leased to others within predetermined periods when underutilized, from both theoretical and practical perspectives. The studied instance subletting service extends and complements the existing instance market of IaaS providers. We identify the unique challenges and opportunities in this new service, and design online auction mechanisms to make allocation and pricing decisions for the instances to be sublet. For static supplies of instances, our mechanism guarantees truthfulness and individual rationality with the best possible competitive ratio. We then incorporate a multi-stage discount strategy to gracefully handle dynamic supplies. Extensive trace-driven simulations show that our service achieves significant performance gains in both cost savings and social welfare. We further pinpoint the challenges in implementing such a service in the real-world system and validate our modeling assumptions through a container-based prototype implemented over Amazon EC2.

Fog and Low Cloud Frequency and Properties from Active-Sensor Satellite Data

An analysis of fog and low cloud properties and distribution is performed using satellite-based LiDAR. Recent years have seen great progress in the remote sensing of fog and low clouds using passive satellite-based sensors. On this basis, maps of fog distribution and frequency as well as baseline climatologies have been constructed. However, no information on fog altitude and vertical extent is available in this way, and fog/low cloud below other clouds cannot be detected in most cases. In this study, ten years of observations by the LiDAR aboard the CALIPSO (Cloud-Aerosol LiDAR and Pathfinder Satellite Observations) platform are used to construct a map and statistical evaluations of fog/low cloud distribution and properties. For the purpose of evaluation, a comparison is made to an evaluation of fog/low cloud distribution in Europe, derived from Meteosat measurements using the Satellite-Based Operation Fog Observation Scheme (SOFOS). Both maps show good agreement in spatial patterns in this region with very diverse fog formation mechanisms. It is found that fog/low cloud layers display distinct spatial differences in terms of geometrical thickness and detection accuracy. The number of fog/low cloud instances missed by passive-sensor retrievals due to multi-layer cloud situations is considerable, with clear regional differences.

A survey on optimal utilization of preemptible VM instances in cloud computing

In this article, an extensive survey on optimal utilization of preemptible instances in cloud is presented. Different techniques used in state-of-the-art research for efficient utilization of spot instances have been classified and categorized in the paper. To the best of our knowledge, this is the first attempt of its kind. With the continuing growths in cloud computing, researchers and business personnels are exploiting the services provided by the cloud computing to reduce their operational cost. Users can share resources in the cloud with the help of virtualization. Virtualization provides abstraction of cloud to the users by hiding the complexity of inherent software and hardware present in the cloud. It increases the likelihood of running multiple operating systems (OSs) on a single physical machine with sharing of hardware resources. Each OS can be considered as a virtual machine (VM) installed on a physical machine. Based on subscription model, VMs can be classified into three types: reserved VMs, on-demand VMs and spot VMs. Spot instances are also known as preemptible VM instances. Spot instances are used as reduced cost resources at the risk of reliability. To utilize spot instances, users have to bid for them. Users will able to get the spot instances only if the biding price is greater than the spot instance price. As soon as the bid price becomes less than the spot price, the cloud provider will revoke the VMs (SIs). This survey aims to find the ways, one can efficiently utilize spot instances for executing the tasks with optimized cost and time.

Accumulating computational resource usage of data analysis workflow to select suitable cloud instance

Accumulating computational resource usage of data analysis workflow to select suitable cloud instance

Cloud-based federated identity for the Internet of Things

The Internet of Things (IoT) has significant security and privacy risks. Recent attacks have shown that not only are many IoT devices at risk of exploit, but those devices can be successfully used to attack wider systems and cause economic damage. Currently, most devices connect to a cloud service that is provided by the manufacturer of the device, offering no choice to move to more secure systems. We outline a proposed model for IoT that allows the identity of users and devices to be federated. Users and devices are issued with secure, random, anonymised identities that are not shared with third parties. We demonstrate how devices can be connected to third-party applications without inherently de-anonymising them. Sensor data and actuator commands are federated through APIs to cloud services. All access to device data and commands is based on explicit consent from users. Each user’s data is handled by a personal cloud instance providing improved security and isolation, as well as providing a trusted intermediary for both devices and cloud services. We demonstrate this model is workable with a prototype system that implements the major features of the model. We present experiment results including performance, energy usage, capacity and cost metrics from the prototype. We compare this work with other related work and outline areas for discussion and future work.

A server consolidation method with integrated deep learning predictor in local storage based clouds

SummaryServer consolidation is one of the critical techniques for energy‐efficiency in cloud data centers. As it is often assumed that cloud service instances (eg, Amazon EC2 instances) utilize the shared storage only. In recent years, however, cloud service providers have been providing local storage for cloud users, since local storage can offer a better performance with identified price. However, these cloud instances usually contain much more data than shared storage cloud instances. Thus, in such local storage based cloud center, the migration cost can be really high and is in dire need of an efficient resource pre‐allocation. If we can predict the resource demand in advance, the migration oscillation will be reduced to minify the migration cost. We have found that there are some related work about server consolidation based on forecasting. Unfortunately, their latest work did not consider the background of “local storage” as we mentioned above. At the same time, some research about local storage did not involve the prediction strategy, which plays a significant part in server consolidation. To address this issue, this paper proposes Losari, a consolidation method, which takes numeric forecasting and local storage architecture into consideration. Losari consolidates servers on the basis of the resource demand predicted value using a statistical learning method. We model the workload from real cloud production environment as a time series. Taking deep learning as a frame of reference, multiple deep belief networks integrated with ARIMA model was trained to study the feature of historical workload. The experimental results have showed that its average predicted error is only 10.7% in the short term, which is much lower than the most common model based on threshold (19.8%) on the same dataset. What is more, the results show that Losari not only simulates the true sequences in high accuracy but also scales the compute resource well, which demonstrated the validity of this integrated deep learning model.

Statistical analysis of Amazon EC2 cloud pricing models

SummaryIn this paper, we conduct statistical analyses for two Amazon cloud pricing models: on demand and spot. On demand cloud instances are charged a fixed price and can only be terminated by the user, with very high availability. On the other hand, spot instances are charged a dynamic price determined by a market‐driven model and can be revoked by the provider when the spot price becomes higher than the user‐defined price, having possibly low availability. Our analysis for on‐demand instances resulted in multiple linear regression equations that represent the influence of characteristics of the processor and RAM memory in the composition of the price of different types of instances available on the Amazon EC2 provider. In order to analyze the Amazon spot pricing, we used time‐smoothed moving averages by 12‐hour periods, aiming to provide a price‐availability trade‐off to the user. Our experiments with spot price histories from September to November 2016 show that the user's bid can be set at 30% of the on‐demand price, with an availability above of 90%, depending on instance type.

Monitoring IaaS using various cloud monitors

In cloud computing, the performance of infrastructure as a service is critical because of its divergence in area. The cloud providers guarantee that the resources will be available around the clock. The providers assure that the period of unavailability of resources is very less. Recently the cloud users have increased rapidly; therefore the providers also have increased, basically increasing the complexity of the infrastructure. This complex infrastructure should be allocated properly to the users and the availability of resources should be notified to the users. So monitoring of these resources constantly is critical. In this analysis, comparison of various monitoring tools in terms of SLA parameters are measured and tabulated. For the comparison, the Amazon cloud instances are monitored with three different monitoring tools like CloudWatch monitoring, IDERA uptime cloud monitor and ManageEngine applications manager. The SLA parameters of IaaS are CPU utilization, network in, network out, disk read, disk write, response time and memory usage. In addition with Amazon instances, servers like Tomcat and data base like PostgreSQL are also monitored and their performance parameters are also analyzed. The instances monitored by cloudwatch monitoring gives twice the range of CPU Utilization than the others. The network data transfer is also high using cloudwatch.

OSPN: Optimal Service Provisioning with Negotiation for Bag-of-Tasks Applications

Cloud service selection is becoming more complex with the arrival of a large number of cloud providers offering various service packages on the market. These cloud service packages are generally provisioned by Spot, On-demand and Reserved Instances. Typically, a user's service requirements contain many independent sub-tasks (Bag-of-Tasks), and have budget limitations and additional constraints. To select reasonable cloud instances to run the user's sub-tasks, we propose a strategy, OSPN (Optimal Service Provisioning with Negotiation), to support the allocation of tasks to services offered by multi-cloud providers. OSPN consists of two phases: in the first phase, a one-to-many parallel Spot Instance pricing negotiation is applied; in the second phase, service provisioning strategy profiles on the three types of cloud instances are calculated. Specifically, the first phase employs an improved double auction in which the price and availability of providers’ instances are taken into account; then the second phase gives the utility Nash equilibrium model and derives the optimal provisioning strategy profiles. The experimental results show that our service provisioning strategy is more cost-effective, namely, the most gains of both the user and providers in the changing scenes, and the least payments of the user than the existing relevant strategies.

Optimal cloud resource provisioning for auto-scaling enterprise applications

Auto-scaling enterprise/workflow systems on cloud needs to deal with both the scaling policy, which determines 'when to scale' and the resource provisioning policy, which determines 'how to scale'. This paper presents a novel resource provisioning policy that can find the most cost optimal setup of variety of instances of cloud that can fulfill incoming workload. All major factors involved in resource amount estimation such as processing power, periodic cost and configuration cost of each instance type, lifetime of each running instance and capacity of clouds are considered in the model. Benchmark experiments were conducted on Amazon cloud and were matched with Amazon AutoScale, using a real load trace and through two main control flow components of enterprise applications, AND and XOR. The experiments showed that the model is plausible for auto-scaling any web/services based enterprise workflow/application on the cloud, along with the effect of individual parameters on the optimal policy.

Optimal cloud resource provisioning for auto-scaling enterprise applications

Auto-scaling enterprise/workflow systems on cloud needs to deal with both the scaling policy, which determines 'when to scale' and the resource provisioning policy, which determines 'how to scale'. This paper presents a novel resource provisioning policy that can find the most cost optimal setup of variety of instances of cloud that can fulfill incoming workload. All major factors involved in resource amount estimation such as processing power, periodic cost and configuration cost of each instance type, lifetime of each running instance and capacity of clouds are considered in the model. Benchmark experiments were conducted on Amazon cloud and were matched with Amazon AutoScale, using a real load trace and through two main control flow components of enterprise applications, AND and XOR. The experiments showed that the model is plausible for auto-scaling any web/services based enterprise workflow/application on the cloud, along with the effect of individual parameters on the optimal policy.

VOLAP: A Scalable Distributed Real-Time OLAP System for High-Velocity Data

This paper presents VelocityOLAP (VOLAP), a distributed real-time OLAP system for high-velocity data. VOLAP makes use of dimension hierarchies, is highly scalable, exploits both multi-core and multi-processor parallelism, and can guarantee serializable execution of insert and query operations. In contrast to other high performance OLAP systems such as SAP HANA or IBM Netezza that rely on vertical scaling or special purpose hardware, VOLAP supports cost-efficient horizontal scaling on commodity hardware or modest cloud instances. Experiments on 20 Amazon EC2 nodes with TPC-DS data show that VOLAP is capable of bulk ingesting data at over 600 thousand items per second, and processing streams of interspersed insertions and aggregate queries at a rate of approximately 50 thousand insertions and 20 thousand aggregate queries per second with a database of 1 billion items. VOLAP is designed to support applications that perform large aggregate queries, and provides similar high performance for aggregations ranging from a few items to nearly the entire database.

Keep the PokerFace on! Thwarting cache side channel attacks by memory bus monitoring and cache obfuscation

Cloud instances are vulnerable to cross-core, cross-VM attacks against the shared, inclusive last-level cache. Automated cache template attacks, in particular, are very powerful as the vulnerabilities do not need to be manually identified. Such attacks can be devised using both the Prime+Probe and the Flush+Reload techniques. In this paper, we present PokerFace, a novel method to identify and mitigate such attacks. This approach allows us to identify suspicious cache accesses automatically, without prior knowledge about the system or access to hardware metrics. PokerFace consists of two components, Poker and Face. Poker executes a memory bus benchmark to measure the available bus bandwidth and derive information about cache accesses and possible side channel attacks. Our experiments with cache attacks show a reduction of up to 14% in the memory bandwidth during the attack. When an attack is detected, Poker triggers Face which performs cache obfuscation. We demonstrate the effectiveness of our approach against keypress logging attacks. We also test it against generic Prime+Probe and Flush+Reload attacks and show that it is practically useful against a variety of cache timing attacks. PokerFace incurs modest overheads (< 8%) and moreover, does not require support from the cloud provider or changes to the hypervisor. Unlike previously proposed techniques, it can be implemented by cloud subscribers.

Price forecasting for spot instances in Cloud computing

Big data applications usually need to rent a large number of virtual machines from Cloud computing providers. As a result of the policies employed by Cloud providers, the prices of spot virtual machine instances behavior stochastically. Spot prices (prices of spot instances) fluctuate greatly or have multiple regimes. Choosing virtual machines according to trends in prices is helpful in decreasing the resource rental cost. Existing price prediction methods are unable to accurately predict prices in these environments. As a result, a dynamic-ARIMA and two markov regime-switching autoregressive model based forecasting methods have been developed in this paper. Experimental results show that the proposals are better than the existing MonthAR in most scenarios.

Improved Cloud resource allocation: how INDIGO-DataCloud is overcoming the current limitations in Cloud schedulers

Performing efficient resource provisioning is a fundamental aspect for any resource provider. Local Resource Management Systems (LRMS) have been used in data centers for decades in order to obtain the best usage of the resources, providing their fair usage and partitioning for the users. In contrast, current cloud schedulers are normally based on the immediate allocation of resources on a first-come, first-served basis, meaning that a request will fail if there are no resources (e.g. OpenStack) or it will be trivially queued ordered by entry time (e.g. OpenNebula). Moreover, these scheduling strategies are based on a static partitioning of the resources, meaning that existing quotas cannot be exceeded, even if there are idle resources allocated to other projects. This is a consequence of the fact that cloud instances are not associated with a maximum execution time and leads to a situation where the resources are under-utilized. These facts have been identified by the INDIGO-DataCloud project as being too simplistic for accommodating scientific workloads in an efficient way, leading to an underutilization of the resources, a non desirable situation in scientific data centers. In this work, we will present the work done in the scheduling area during the first year of the INDIGO project and the foreseen evolutions.

Simulation of a workflow execution as a real Cloud by adding noise

Cloud computing provides a cheap and elastic platform for executing large scientific workflow applications, but it rises two challenges in prediction of makespan (total execution time): performance instability of Cloud instances and variant scheduling of dynamic schedulers. Estimating the makespan is necessary for IT managers in order to calculate the cost of execution, for which they can use Cloud simulators. However, the ideal simulated environment produces the same output for the same workflow schedule and input parameters and thus can not reproduce the Cloud variant behavior. In this paper, we define a model and a methodology to add a noise to the simulation in order to equalise its behavior with the Clouds’ one. We propose several metrics to model a Cloud fluctuating behavior and then by injecting them within the simulator, it starts to behave as close as the real Cloud. Instead of using a normal distribution naively by using mean value and standard deviation of workflow tasks’ runtime, we inject two noises in the tasks’ runtime: noisiness of tasks within a workflow (defined as average runtime deviation) and noisiness provoked by the environment over the whole workflow (defined as average environmental deviation). In order to measure the quality of simulation by quantifying the relative difference between the simulated and measured values, we introduce the parameter inaccuracy. A series of experiments with different workflows and Cloud resources were conducted in order to evaluate our model and methodology. The results show that the inaccuracy of the makespan’s mean value was reduced up to 59 times compared to naively using the normal distribution. Additionally, we analyse the impact of particular workflow and Cloud parameters, which shows that the Cloud performance instability is simulated more correctly for small instance type (inaccuracy of up to 11.5%), instead of medium (inaccuracy of up to 35%), regardless of the workflow. Since our approach requires collecting data by executing the workflow in the Cloud in order to learn its behavior, we conduct a comprehensive sensitivity analysis. We determine the minimum amount of data that needs to be collected or minimum number of test cases that needs to be repeated for each experiment in order to get less than 12% inaccuracy for our noising parameter. Additionally, in order to reduce the number of experiments and determine the dependency of our model against Cloud resource and workflow parameters, the conducted comprehensive sensitivity analysis shows that the correctness of our model is independent of workflow parallel section size. With our sensitivity analysis, we show that we can reduce the inaccuracy of the naive approach with only 40% of total number of executions per experiment in the learning phase. In our case, 20 executions per experiment instead of 50, and only half of all experiments, which means down to 20%, i.e. 120 test cases instead of 600.

A Compact Hybrid Evolutionary Algorithm for Large Scale Instance Matching in Linked Open Data Cloud

Establishing correct links among the coreference ontology instances is critical to the success of Linked Open Data (LOD) cloud. However, because of the high level heterogeneity and large scale instance set, matching the coreference instances in LOD cloud is an error prone and time consuming task. To this end, in this work, we present an asymmetrical profile-based similarity measure for instance matching task, construct new optimal models for schema-level and instance-level matching problems, and propose a compact hybrid evolutionary algorithm based ontology matching approach to solve the large scale instance matching problem in LOD cloud. Finally, the experimental results of comprising our approach with the states of the art systems on the instance matching track of OAEI 2015 and real-world datasets show the effectiveness of our approach.

Cloud Forensics : Isolating Cloud Instance

Abstract—Cloud computing has been the trending model for storing, accessing and modifying the data over the Internet in the recent years. Rising use of the cloud has generated a new concept related to the cloud which is cloud forensics. Cloud forensics can be defined as investigating for evidence over the cloud, so it can be viewed as a combination of both cloud computing and digital forensics. Many issues of applying forensics in the cloud have been addressed. Isolating the location of the incident has become an essential part of forensic process. This is done to ensure that evidence will not be modified or changed. Isolating an instant in the cloud computing has become even more challenging, due to the nature of the cloud environment. In the cloud, the same storage or virtual machine have been used by many users. Hence, the evidence is most likely will be overwritten and lost. The proposed solution in this paper is to isolate a cloud instance. This can be achieved by marking the instant that reside in the servers as "Under Investigation". To do so, cloud file system must be studied. One of the well-known file systems used in the cloud is Apache Hadoop Distributed File System (HDFS). Thus, in this paper the methodology used for isolating a cloud instance would be based on the HDFS architecture.&#x0D; Keywords: cloud computing; digital forensics; cloud forensics

CloudNeo: a cloud pipeline for identifying patient-specific tumor neoantigens.

SummaryWe present CloudNeo, a cloud-based computational workflow for identifying patient-specific tumor neoantigens from next generation sequencing data. Tumor-specific mutant peptides can be detected by the immune system through their interactions with the human leukocyte antigen complex, and neoantigen presence has recently been shown to correlate with anti T-cell immunity and efficacy of checkpoint inhibitor therapy. However computing capabilities to identify neoantigens from genomic sequencing data are a limiting factor for understanding their role. This challenge has grown as cancer datasets become increasingly abundant, making them cumbersome to store and analyze on local servers. Our cloud-based pipeline provides scalable computation capabilities for neoantigen identification while eliminating the need to invest in local infrastructure for data transfer, storage or compute. The pipeline is a Common Workflow Language (CWL) implementation of human leukocyte antigen (HLA) typing using Polysolver or HLAminer combined with custom scripts for mutant peptide identification and NetMHCpan for neoantigen prediction. We have demonstrated the efficacy of these pipelines on Amazon cloud instances through the Seven Bridges Genomics implementation of the NCI Cancer Genomics Cloud, which provides graphical interfaces for running and editing, infrastructure for workflow sharing and version tracking, and access to TCGA data.Availability and implementationThe CWL implementation is at: https://github.com/TheJacksonLaboratory/CloudNeo. For users who have obtained licenses for all internal software, integrated versions in CWL and on the Seven Bridges Cancer Genomics Cloud platform (https://cgc.sbgenomics.com/, recommended version) can be obtained by contacting the authors.Supplementary information Supplementary data are available at Bioinformatics online.