Scientific Workflows In Cloud Research Articles

Securing the Intermediate Data of Scientific Workflows in Clouds With ACISO

Many scientific workflow applications are moving to clouds. A scientific workflow is a complicated scientific computing task consisting of many sub-tasks, and each sub-task execution can generate the intermediate data used for the successor sub-task execution. The correct execution of scientific workflows depends on the security of the intermediate data, which is transmitted frequently between virtual machines during the process of the workflow execution. In multi-tenant clouds, the intermediate data contains three attributes: availability, confidentiality and integrity. If the intermediate data is lost, stolen, or tampered with by malicious tenants, the intermediate data’s attribute will be damaged, causing workflow interruption, the leakage of secret information or incorrect workflow results. For these problems, we propose ACISO scheme to secure the intermediate data by improving its availability, confidentiality, and integrity. In the scheme, availability, confidentiality and integrity strategy pools are constructed by various erasure codes, encryption algorithms and hash functions, respectively. Then we present a security strategy optimal allocation model named SSOA, which aims to maximize the overall intermediate data security strength while meeting the constraints of the workflow makespan and storage overhead. Normally, a scientific workflow contains a large number of the intermediate data, so solving this model is NP hard. Therefore, we propose a heuristic solution to solve SSOA. The simulation results show that ACISO can effectively improve the availability, confidentiality, and integrity of the intermediate data of the scientific workflows.

Ориентированное на данные планирование с применением отказоустойчивого метода динамической кластеризации для поддержки потоков научных работ в облаках

Ориентированное на данные планирование с применением отказоустойчивого метода динамической кластеризации для поддержки потоков научных работ в облаках

On efficient resource use for scientific workflows in clouds

The abundance of cloud resources has enabled not only web applications, but also scientific applications to easily scale to meet their objectives, such as performance and costs. However, due to the complex and large-scale nature of scientific workflows, the decision on such scaling (resource management) is much complicated often resulting in inefficient use of resources. In this paper, we present RDAS+ as a resource demand aware scheduling algorithm to optimize resource efficiency for the execution of scientific workflows in clouds. RDAS+ maximizes resource utilization by allocating the minimum number of resources (virtual machines or VMs in clouds) with little sacrifice of completion time (makespan). This optimization eventually leads to cost efficiency for pay-per-use cloud resources. RDAS+ consists of partitioning, resource allocation and task scheduling steps to realize such optimization. We have evaluated RDAS+ using five types of real-world scientific workflows in comparison with three existing algorithms. Our experimental results confirm our claims on achieving resource efficiency. In particular, the average rate of cost savings (32%) outweighs makespan increase (11%). Although these two performance metrics are incompatible, the trade-off RDAS+ optimizes shows significant benefit particularly in clouds.

Cost Effective Hybrid Genetic Algorithm for Scheduling Scientific Workflows in Cloud under Deadline Constraint

Cost Effective Hybrid Genetic Algorithm for Scheduling Scientific Workflows in Cloud under Deadline Constraint

Smart PSO-based secured scheduling approaches for scientific workflows in cloud computing

Owing to its manifold advantages in adapting cloud computing for real-world scientific workflow applications, we intend to use cloud computing for executing the scientific workflows. In the present work, we aim for scheduling the workflow in the scalable resources in the cloud. In general, security is a vital challenge in cloud and so we include security constraints into our optimization model. The main objective of our work is to find an optimized schedule having minimum makespan and cost and by satisfying security demand constraint. The users can submit their security demand to the cloud provider during negotiation. The workflow is initially scheduled with list-based heuristics, which is then optimized by Particle Swarm Optimization (PSO). Thus we device a Smart Particle Swarm Optimization (SPSO)-based secured scheduling to find the optimized schedule with minimum makespan and cost. The proposed method is capable of assigning the task in the scientific workflows to the best suitable virtual machine in the cloud. Hence, the resource allocation is addressed as well by our method. Besides, a variant of PSO algorithm called Variable Neighbourhood PSO is also experimented to overcome the local optima problem. Our experimental results show that the scheduled workflows with assured security are yielding better makespan than existing methods with minimum iterations, which is well suited for cloud environment.

A Declarative Optimization Engine for Resource Provisioning of Scientific Workflows in Geo-Distributed Clouds

Geo-distributed clouds are becoming increasingly popular for cloud providers, and data centers with different regions often offer different prices, even for the same type of virtual machines. Resource provisioning in geo-distributed clouds is an important and complicated problem for budget and performance optimizations of scientific workflows. Scientists are facing the complexities resulted from various cloud offerings in the geo-distributed settings, severe cloud performance dynamics and evolving user requirements on performance and cost. To address those complexities, we propose a declarative optimization engine named Geco for resource provisioning of scientific workflows in geo-distributed clouds. Geco allows users to specify their workflow optimization goals and constraints of specific problems with an extended declarative language. We propose a novel probabilistic optimization approach for evaluating the declarative optimization goals and constraints to address the cloud dynamics. Additionally, we develop runtime optimizations to more effectively utilize the cloud resources at runtime. To accelerate the solution finding, Geco leverages the power of GPUs to find the solution in a fast and timely manner. Our evaluations with four common workflow provisioning problems demonstrate that, Geco is able to achieve more effective performance/cost optimizations in geo-distributed cloud environments than the state-of-the-art approaches.

A taxonomy and survey on scheduling algorithms for scientific workflows in IaaS cloud computing environments

SummaryLarge‐scale scientific problems are often modeled as workflows. The ever‐growing data and compute requirements of these applications has led to extensive research on how to efficiently schedule and deploy them in distributed environments. The emergence of the latest distributed systems paradigm, cloud computing, brings with it tremendous opportunities to run scientific workflows at low costs without the need of owning any infrastructure. It provides a virtually infinite pool of resources that can be acquired, configured, and used as needed and are charged on a pay‐per‐use basis. However, along with these benefits come numerous challenges that need to be addressed to generate efficient schedules. This work identifies these challenges and studies existing algorithms from the perspective of the scheduling models they adopt as well as the resource and application model they consider. A detailed taxonomy that focuses on features particular to clouds is presented, and the surveyed algorithms are classified according to it. In this way, we aim to provide a comprehensive understanding of existing literature and aid researchers by providing an insight into future directions and open issues.

Implementing Enhanced ICPCP Algorithm with Task Replication in Public Cloud

Cloud Computing has Large Scale Distributed Infrastructure which is accessible and scalable infrastructure. Cloud computing provides a pay as you go model in which the user has to pay for the services he uses. One of the characteristic of cloud is elasticity in which resources can be dynamically increases or decreases as per user requirement. The goal of this project is to execute the scientific workflows in public cloud within user define deadline and smallest possible cost. The deadline of the project can be meeting by provisioning more virtual machines that required. The algorithm Enhanced ICPCP uses the concept partial critical path which is defined in the ICPCP. The simulation result shows the algorithm reduces the execution time of different scientific workflows simulated using the cloudsim. General Terms Cloud Computing, Scheduling, Scientific Workflow.

Catfish-PSO based scheduling of scientific workflows in IaaS cloud

Cloud computing is a technology wherein a network of remote servers is used to process large amount of data in real-time. The servers and data sources may be located in geographically distant regions. Scheduling of workflows is one of the major challenging issues in cloud computing. Workflows are used to express a wide variety of applications including scientific computing and multi-tier web applications. The Workflow scheduling problem is known to be NP-complete. No known traditional scheduling algorithm is able to provide an optimal solution in polynomial time for NP-complete problems. So, researchers rely on heuristics and meta-heuristics to achieve the most efficient solution. In this paper, a workflow scheduling algorithm is proposed to schedule large scientific workflows that are to be executed on IaaS clouds. The workflow scheduling algorithm generates a schedule with the task-to-resource mapping. The metaheuristic Catfish particle swarm optimization (C-PSO) technique is used to select the best schedule with the least makespan and execution cost. The performance of C-PSO is then compared with traditional PSO. The algorithm is simulated on the WorkFlowSim Simulator, an extension of CloudSim simulator. The solution is tested for different types of scientific workflows like Montage, Epigenome, CyberShake and Inspiral. It is observed from the experimental results that C-PSO gives better performance than traditional PSO in terms of execution cost and makespan.

A Dynamic Cloud Dimensioning Approach for Parallel Scientific Workflows: a Case Study in the Comparative Genomics Domain

Usually, scientists need to execute experiments that demand high performance computing environments and parallel techniques. This is the scenario found in many bioinformatics experiments modeled as scientific workflows, such as phylogenetic and phylogenomic analyses. To execute these experiments, scientists have adopted virtual machines (VMs) instantiated in clouds. Estimating the number of VMs to instantiate is a crucial task to avoid negative impacts on the execution performance and on the financial costs with under or overestimations. Previously, the necessary number of VMs to execute bioinformatics workflows have been estimated by a GRASP heuristic and have been coupled to a Cloud-based Parallel Scientific Workflow Management System. Although this work was a step forward, this approach only provided a static dimensioning. If the characteristics of the environment change (processing capacity, network speed), this static dimensioning may not be suitable. In this way, it is of interest that the dimensioning is adjusted at runtime. To achieve this, we developed a novel framework for monitoring and dynamically dimensioning resources during the execution of parallel scientific workflows in clouds, called Dynamic Dimensioning of Cloud Computing Framework (DDC-F). We have evaluated DDC-F in real executions of bioinformatics workflows. Experiments showed that DDC-F is able to efficiently calculate the number of VMs necessary to execute bioinformatics workflows of Comparative Genomics (CG), also reducing the financial costs, when compared with other works of the related literature.

A Survey on Data Placement Strategies for Cloud based Scientific Workflows

workflows perform computations exceeding single workstation's capabilities. When running such data intensive workflows in the cloud distributed across several physical locations, the execution time and the resource utilization efficiency highly depends on the initial placement and distribution of the input datasets across these multiple virtual machines in the Cloud. The ideal data placement scheme optimizes the execution of the data intensive scientific workflows in cloud by assigning the tasks to the execution site in such a way that the file transfers and the cost associated are reduced. Several data placement strategies in cloud based scientific workflows are reviewed. A data placement scheme which uses big data to improve the performance and also the data movement cost is studied. BDAP (Big Data Placement strategy), improves workflow performance by minimizing data movement across multiple virtual machines. Keywordscomputing, Big data, Scientific workflow, Data

Task scheduling strategy based on data replication in scientific Cloud workflows

The invention of Cloud computing as a new model of service provisioning in distributed systems encourages re- searchers to investigate its benefits and drawbacks in executing applications. In recent years, Cloud computing is fast evolving as the target platform for such applications among researchers. Furthermore, new pricing models have been pioneered by Cloud providers that allow users to provision resources and to use them in an efficient manner with significant cost reductions. Ap- proaches for scheduling and data placement is often highly correlated, which take into account a few factors at the same time, and what are the most often adapted to applications data medium and therefore doesn't go to scale. In this work, we propose an optimization approach that takes into account an effective data placement and scheduling of tasks grouped based on data replication in scientific Cloud environments. This proposed approach improve data placement and minimize response time due to scheduling tasks to data centers that contain the majority of the required data.

A balanced scheduler with data reuse and replication for scientific workflows in cloud computing systems

Cloud computing provides substantial opportunities to researchers who demand pay-as-you-go computing systems. Although cloud provider (e.g., Amazon Web Services) and application provider (e.g., biologists, physicists, and online gaming companies) both have specific performance requirements (e.g. application response time), it is the cloud scheduler’s responsibility to map the application to underlying cloud resources. This article presents a Balanced and file Reuse–Replication Scheduling (BaRRS) algorithm for cloud computing environments to optimally schedule scientific application workflows. BaRRS splits scientific workflows into multiple sub-workflows to balance system utilization via parallelization. It also exploits data reuse and replication techniques to optimize the amount of data that needs to be transferred among tasks at run-time. BaRRS analyzes the key application features (e.g., task execution times, dependency patterns and file sizes) of scientific workflows for adapting existing data reuse and replication techniques to cloud systems. Further, BaRRS performs a trade-off analysis to select the optimal solution based on two optimization constraints: execution time and monetary cost of running scientific workflows. BaRRS is compared with a state-of-the-art scheduling approach; experiments prove its superior performance. Experiments include four well known scientific workflows with different dependency patterns and data file sizes. Results were promising and also highlighted most critical factors affecting execution of scientific applications on clouds.

Necessary and sufficient checkpoint selection for temporal verification of high-confidence cloud workflow systems

On-time completion is an important temporal QoS (Quality of Service) dimension and one of the fundamental requirements for high-confidence workflow systems. In recent years, a workflow temporal verification framework, which generally consists of temporal constraint setting, temporal checkpoint selection, temporal verification, and temporal violation handling, has been the major approach for the high temporal QoS assurance of workflow systems. Among them, effective temporal checkpoint selection, which aims to timely detect intermediate temporal violations along workflow execution plays a critical role. Therefore, temporal checkpoint selection has been a major topic and has attracted significant efforts. In this paper, we will present an overview of work-flow temporal checkpoint selection for temporal verification. Specifically, we will first introduce the throughput based and response-time based temporal consistency models for business and scientific cloud workflow systems, respectively. Then the corresponding benchmarking checkpoint selection strategies that satisfy the property of“necessity and sufficiency”are presented. We also provide experimental results to demonstrate the effectiveness of our checkpoint selection strategies, and finally points out some possible future issues in this research area.

End-to-End Delay Minimization for Scientific Workflows in Clouds under Budget Constraint

Next-generation e-Science features large-scale, compute-intensive workflows of many computing modules that are typically executed in a distributed manner. With the recent emergence of cloud computing and the rapid deployment of cloud infrastructures, an increasing number of scientific workflows have been shifted or are in active transition to cloud environments. As cloud computing makes computing a utility, scientists across different application domains are facing the same challenge of reducing financial cost in addition to meeting the traditional goal of performance optimization. We develop a prototype generic workflow system by leveraging existing technologies for a quick evaluation of scientific workflow optimization strategies. We construct analytical models to quantify the network performance of scientific workflows using cloud-based computing resources, and formulate a task scheduling problem to minimize the workflow end-to-end delay under a user-specified financial constraint. We rigorously prove that the proposed problem is not only NP-complete but also non-approximable. We design a heuristic solution to this problem, and illustrate its performance superiority over existing methods through extensive simulations and real-life workflow experiments based on proof-of-concept implementation and deployment in a local cloud testbed.

Autonomic fault tolerant scheduling approach for scientific workflows in Cloud computing

Autonomic fault tolerant scheduling is now a mandatory approach for the execution of performance-motivated Cloud applications such as scientific workflows. Since concurrent engineering is strongly associated with scientific workflows, an efficient scheduling for scientific workflows can have major impact on the performance of concurrent systems and engineering applications in Cloud computing. To facilitate the execution of concurrent tasks in scientific workflows, Cloud providers entail efficient scheduling heuristics and fault tolerant approaches. The work presented in this article formulates an effort focusing on this research problem to design an autonomic fault tolerant scheduling approach for scientific workflow applications. First, hybrid heuristic has been proposed to schedule scientific workflows effectively. Second, fault tolerant technique has been implemented using virtual machine migration approach that migrates the virtual machine automatically in case of task failure occurrences due to the overutilization of resources. Furthermore, the proposed approach has been validated through performance evaluation parameters using CloudSim and WorkflowSim toolkits. The simulation results demonstrate the effectiveness of the proposed approach to improve the performance of scientific workflows by appreciably reducing total mean execution time, standard deviation time, and makespan.

Scheduling of scientific workflows in cloud with replication

Cloud infrastructure is rapidly scalable, accessible and also it is a pay as we use model. Due to the elastic nature of cloud, the resource provisioning can be done dynamically according to the requirements of the workflow application. We aim at executing the scientific workflows within the user-defined deadline, with the smaller possible cost by reducing the total execution time of applications. The execution time of the task is reduced by proper provisioning and scheduling of all tasks in the scientific workflows. The proposed algorithm RIPCP (Replication based IC-PCP) uses the principle of IC-PCP algorithm for finding the Partial Critical Path (PCP) of the tasks and includes the task replication for better make span. Three different Scientific Workflows are simulated in different characteristics of virtual machines to demonstrate that the proposed algorithm reduces the total execution time of the applications.

Leveraging Workflows and Clouds for a Multi-frontal Solver for Finite Element Meshes

Scientific workflows in clouds have been successfully used for automation of large-scale computations, but so far they were applied to the loosely-coupled problems, where most workflow tasks can be processed independently in parallel and do not require high volume of communication. The multi-frontal solver algorithm for finite element meshes can be represented as a workflow, but the fine granularity of resulting tasks and the large communication to computation ratio makes it hard to execute it efficiently in loosely-coupled environments such as the Infrastructure-as-a-Service clouds. In this paper, we hypothesize that there exists a class of meshes that can be effectively decomposed into a workflow and mapped onto a cloud infrastructure. To show that, we have developed a workflow-based multi-frontal solver using the HyperFlow workflow engine, which comprises workflow generation from the elimination tree, analysis of the workflow structure, task aggregation based on estimated computation costs, and distributed execution using a dedicated worker service that can be deployed in clouds or clusters. The results of our experiments using the workflows of over 10,000 tasks indicate that after task aggregation the resulting workflows of over 100 tasks can be efficiently executed, and the overheads are not prohibitive. These results lead us to conclusions that our approach is feasible and gives prospects for providing a generic workflow-based solution using clouds for problems typically considered as requiring HPC infrastructure.

Optimizing virtual machine allocation for parallel scientific workflows in federated clouds

Cloud computing has established itself as an interesting computational model that provides a wide range of resources such as storage, databases and computing power for several types of users. Recently, the concept of cloud computing was extended with the concept of federated clouds where several resources from different cloud providers are inter-connected to perform a common action (e.g. execute a scientific workflow). Users can benefit from both single-provider and federated cloud environment to execute their scientific workflows since they can get the necessary amount of resources on demand. In several of these workflows, there is a demand for high performance and parallelism techniques since many activities are data and computing intensive and can execute for hours, days or even weeks. There are some Scientific Workflow Management Systems (SWfMS) that already provide parallelism capabilities for scientific workflows in single-provider cloud. Most of them rely on creating a virtual cluster to execute the workflow in parallel. However, they also rely on the user to estimate the amount of virtual machines to be allocated to create this virtual cluster. Most SWfMS use this initial virtual cluster configuration made by the user for the entire workflow execution. Dimensioning the virtual cluster to execute the workflow in parallel is then a top priority task since if the virtual cluster is under or over dimensioned it can impact on the workflow performance or increase (unnecessarily) financial costs. This dimensioning is far from trivial in a single-provider cloud and specially in federated clouds due to the huge number of virtual machine types to choose in each location and provider. In this article, we propose an approach named GraspCC-fed to produce the optimal (or near-optimal) estimation of the amount of virtual machines to allocate for each workflow. GraspCC-fed extends a previously proposed heuristic based on GRASP for executing standalone applications to consider scientific workflows executed in both single-provider and federated clouds. For the experiments, GraspCC-fed was coupled to an adapted version of SciCumulus workflow engine for federated clouds. This way, we believe that GraspCC-fed can be an important decision support tool for users and it can help determining an optimal configuration for the virtual cluster for parallel cloud-based scientific workflows.

Meeting Deadlines of Scientific Workflows in Public Clouds with Tasks Replication

The elasticity of Cloud infrastructures makes them a suitable platform for execution of deadline-constrained workflow applications, because resources available to the application can be dynamically increased to enable application speedup. Existing research in execution of scientific workflows in Clouds either try to minimize the workflow execution time ignoring deadlines and budgets or focus on the minimization of cost while trying to meet the application deadline. However, they implement limited contingency strategies to correct delays caused by underestimation of tasks execution time or fluctuations in the delivered performance of leased public Cloud resources. To mitigate effects of performance variation of resources on soft deadlines of workflow applications, we propose an algorithm that uses idle time of provisioned resources and budget surplus to replicate tasks. Simulation experiments with four well-known scientific workflows show that the proposed algorithm increases the likelihood of deadlines being met and reduces the total execution time of applications as the budget available for replication increases.