Cloud Workflow Research Articles

Energy-Saving Cloud Workflow Scheduling Based on Optimistic Cost Table

Energy-Saving Cloud Workflow Scheduling Based on Optimistic Cost Table

A parallel multi-objective genetic algorithm for scheduling scientific workflows in cloud computing

Recent developments in cloud computing have made it a powerful solution for executing large-scale scientific problems. The complexity of scientific workflows demands efficient utilization of cloud resources to satisfy user requirements. Scheduling of scientific workflows in a cloud environment is a challenge for researchers. The problem is considered as NP-hard. Some constraints such as a heterogeneous environment, dependencies between tasks, quality of service and user deadlines make it difficult for the scheduler to fully utilize available resources. The problem has been extensively studied in the literature. Different researchers have targeted different parameters. This article presents a multi-objective scheduling algorithm for scheduling scientific workflows in cloud computing. The solution is based on genetic algorithm that targets makespan, monetary cost, and load balance. The proposed algorithm first finds the best solution for each parameter. Based on these solutions, the algorithm finds the superbest solution for all parameters. The proposed algorithm is evaluated with benchmark datasets and comparative results with the standard genetic algorithm, particle swarm optimization, and specialized scheduler are presented. The results show that the proposed algorithm achieves an improvement in makespan and reduces the cost with a well load balanced system.

Intelligent cloud workflow management and scheduling method for big data applications

With the application and comprehensive development of big data technology, the need for effective research on cloud workflow management and scheduling is becoming increasingly urgent. However, there are currently suitable methods for effective analysis. To determine how to effectively manage and schedule smart cloud workflows, this article studies big data from various aspects and draws the following conclusions: Compared with the original JStorm system, the response time is shortened by a maximum of 58.26% and an average of 23.18%, CPU resource utilization is increased by a maximum of 17.96% and an average of 11.39%, and memory utilization increased by a maximum of 88.7% and an average of 71.16%. In terms of optimizing the dynamic combination of web services, the overall performance of both the MOACO and CCA algorithms is better than that of the GA algorithm, and the average performance of the MOACO algorithm is better than that of the CCA algorithm. This paper also proposes a cloud workflow scheduling strategy based on an intelligent algorithm and realizes the two-tier scheduling of cloud workflow tasks by adjusting the combination strategy for cloud service resources. We have studied three representative intelligent algorithms (ACO, PSO and GA) and improved them for scheduling optimization. It can be clearly seen that in the same scenario, the optimal values of the different algorithms vary greatly for different test cases. However, the optimal solution curve is substantially consistent with the trend of the mean curve.

Cost and makespan scheduling of workflows in clouds using list multiobjective optimization technique

Abstract Highly scalable resource supply capacity of cloud computing has greatly improved the execution speed of workflow applications, however, traditional workflow scheduling algorithms which focus on the optimization of makespan (execution time) of workflows, become inappropriate for the design of large-scale workflow systems. Workflow scheduling in cloud computing is particularly a multiobjective optimization problem, in which many critical issues besides the execution time of workflows should be taken into account. Although many heuristics and meta-heuristics have been proposed to solve this problem, most of them cannot produce satisfactory cost-makespan tradeoffs and have a long time overhead. In this paper, we propose an efficient heuristic named CMSWC (Cost and Makespan Scheduling of Workflows in the Cloud) to solve the workflow scheduling problem, by simultaneously minimizing cost and makespan of workflows. CMSCW follows a two-phase list scheduling philosophy: ranking and mapping. Furthermore, CMSCW incorporates with three designs specifically for the multiobjective challenges: (i) The mapping phase is designed to avoid exploring useless resources for tasks, which significantly narrows down the search space. (ii) A new method is proposed to select non-dominated solutions, by combining the quick non-dominated sorting approach and Shift-Based Density Estimation (SDE) based crowding distance. (iii) Several elitist study strategies are designed to make solutions close to the true Pareto front as well as avoid trapping into local optimum. Extensive experiments on real-life workflows demonstrate that our approach can generate better cost-makespan tradeoff fronts than that of several state-of-the-art approaches.

PFPMine: A parallel approach for discovering interacting data entities in data-intensive cloud workflows

With the evolution of cloud computing, communities and companies deployed their workflows on cloud to support end-to-end business processes that are usually syndicated with other external services. To improve the efficiency of the system as well as reducing energy consumption, data placement and backup strategies should be carefully designed. One of the most challenging problems is the discovery of interacting data entities in date-intensive workflows. To tackle this challenge, this paper presents a frequent pattern-based approach named FPMine for interacting data entity discovery in cloud workflows. A direct discriminative mining algorithm is first proposed to determine the minimum support threshold, based on which FP-tree is constructed to formulate the frequent item pairs. Next, FP-matrix is applied to avoid traversing the FP-trees during data entity discovery, and a pruning approach is introduced to reduce the redundancy of frequent item pairs. Furthermore, we propose a parallel data entity mining algorithm using MapReduce framework, namely PFPMine, and then design a primitive data placement and backup strategy. Finally, we evaluate the efficiency of our approach by experiments using real-life data, based on which we show that our approach can facilitate the discovery of interacting data entities with efficiency for cloud workflows. Comparing with traditional FP-growth approach, we pay only a multiplicative factor for making our approach able to extract fine-grained frequent item pairs rather than frequent patterns, which can bring significant advantages to data placement. After parallelization, the PFPMine algorithm performs better with high efficiency for both sparse datasets and dense datasets than FP-growth. The results show that PFPMine can reduce the running time by at least 25%, and preforms with significantly higher efficiency than FP-growth approach.

A hybrid genetic algorithm for scientific workflow scheduling in cloud environment

Nowadays, we live an unprecedented evolution in cloud computing technology that coincides with the development of the vast amount of complex interdependent data which make up the scientific workflows. All these circumstances developments have made the issue of workflow scheduling very important and of absolute priority to all overlapping parties as the provider and customer. For that, work must be focused on finding the best strategy for allocating workflow tasks to available computing resources. In this paper, we consider the scientific workflow scheduling in cloud computing. The main role of our model is to optimize the time needed to run a set of interdependent tasks in cloud and in turn reduces the computational cost while meeting deadline and budget. To this end, we offer a hybrid approach based on genetic algorithm for modelling and optimizing a workflow-scheduling problem in cloud computing. The heterogeneous earliest finish time (HEFT), an heuristic model, intervenes in the generation of the initial population. Based on results obtained from our simulations using real-world scientific workflow datasets, we demonstrate that the proposed approach outperforms existing HEFT and other strategies examined in this paper. In other words, experiments show high efficiency of our proposed approach, which makes it potentially applicable for cloud workflow scheduling. For this, we develop a GA-based module that was integrated to the WorkflowSim framework based on CloudSim.

An Algorithmic Analysis on Load Balancing and Workflow Scheduling in Cloud

Cloud computing is a paradigm for facilitating omnipresent, comfortable, on-demand network access to shared provisions of configurable computing resources like servers, applications, networks, and services that can be quickly provisioned and discharged with minimal management effort or service provider intercommunication. Data centers are the set of hosts that runs Virtual Machines. The broker submits the requests which are provisioned to the data centers that submit the task. The task or an application that runs is known as cloudlet. The term “Cloud computing” is to deliver the hosted services on the internet. Each “cloudlet” requires one or more computing resources from the cloud to complete its job. The executions of these sub-tasks use a scheduling algorithm in a particular fashion. Hence, one of the most important issues is that executing a large number of workflow in cloud environment and consumption of resources. The work proposes literature about the scheduling algorithm that resolves the conflict in the allocation of resources and manages the resource provisioning at dynamic to each task in any workflow. This paper proposes an analysis of proposed schemes with different workflow by executing the scheduling algorithm for the cloud workflow. This paper is presented with their objectives, parameters, and limitations and finally concluded with research directions.

Logchain: Cloud workflow reconstruction & troubleshooting with unstructured logs

Cloud-based virtualization has become a key part of building distributed applications. One of its many benefits is the ability to dynamically manage system capacity by creating, deleting and migrating virtual machines (VMs) on-demand. This management process, however, depends on complex pipelines, involving multiple services invocations across distributed nodes. This makes troubleshooting and debugging difficult, as these complex pipelines lack an integrated logging system. Instead, each service generates independent and unstructured log messages without the ability to link logs into a single integrated workflow. We present LogChain, a tool that gathers and processes distributed unstructured logs to diagnose failures in cloud management tasks. It contains three key functions: (i) It infers task workflows from distributed unstructured logs; (ii) it labels these workflows with the tasks that triggered them; and (iii) it diagnoses potential failures in the workflow’s execution, to support administrator with troubleshooting. We evaluate LogChain with realistic workloads, and show that it exceeds the state-of-the-art in terms of performance and accuracy.

Runtime Verification of Business Cloud Workflow Temporal Conformance

Business cloud workflows are often designed with multiple time constraints for timely response to business requests. To ensure on-time completion of workflow instances, workflow temporal conformance state needs to be constantly monitored and verified at runtime. Considering the fact that there are a large number of workflow instances running in a parallel fashion in many business scenarios, conventional verification approaches for time-related properties using such as temporal logic or timed Petri nets are not feasible due to the limitation of low efficiency at runtime. To address this issue, we propose a new approach to automated runtime verification of temporal conformance for parallel workflow instances in a cloud environment. Instead of using response time to verify temporal conformance of every single workflow as in conventional strategies, workflow throughput is employed as the performance measurement to efficiently monitor a large number of parallel workflow instances. On this basis we present a novel conformance verification strategy. This strategy considers the effect of time delay propagation in the cloud workflow systems to accurately verify workflow runtime temporal conformance. Our verification strategy is implemented in a prototype cloud workflow system and the evaluation results show that it outperforms the state-of-the-art workflow temporal verification strategy.

Statistical Model Checking-Based Evaluation and Optimization for Cloud Workflow Resource Allocation

Due to the existence of resource variations, it is very challenging for Cloud workflow resource allocation strategies to guarantee a reliable Quality of Service (QoS). Although dozens of resource allocation heuristics have been developed to improve the QoS of Cloud workflow, it is hard to predict their performance under variations because of the lack of accurate modeling and evaluation methods. So far, there is no comprehensive approach that can quantitatively reason the capability of resource allocation strategies or enable the tuning of parameters to optimize resource allocation solutions under variations. To address the above problems, this paper proposes a novel framework that can evaluate and optimize resource allocation strategies effectively and quantitatively. By using the statistical model checker UPPAAL-SMC and supervised learning approaches, our framework can: i) conduct complex QoS queries on resource allocation instances considering resource variations; ii) make quantitative and qualitative comparisons among resource allocation strategies; iii) enable the tuning of parameters to improve the overall QoS; and iv) support the quick optimization of overall workflow QoS under customer requirements and resource variations. The experimental results demonstrate that our automated framework can support both the Service Level Agreement (SLA) negotiation and workflow resource allocation optimization efficiently.

A Data and Task Co-Scheduling Algorithm for Scientific Cloud Workflows

Cloud computing has emerged as a promising computational infrastructure for cost-efficient workflow execution by provisioning on-demand resources in a pay-as-you-go manner. While scientific workflows require accessing community-wide resources, they usually need to be performed in collaborative cloud environments composed of multiple datacenters. Although such environments facilitate scientific collaboration, the movements of input and intermediate datasets across geographically distributed datacenters may cause intolerable latency that would hinder efficient execution of large-scale data-intensive scientific workflows. To address the problem, in this article we propose a novel multi-level K-cut graph partitioning algorithm to minimize the volume of data transfer across datacenters while satisfying load balancing and fixed data constraints. The algorithm first contracts the fixed input datasets in the same datacenter and their consuming tasks, and coarsens the contracted graph to a predefined scale in a level-by-level manner. Then, a K-cut algorithm is used to partition the resulted graph into K parts such that the cut size is minimized. After that, the partitioned graph is projected back to the original workflow graph, during which the load balancing constraint is maintained. We evaluate our algorithm using three real-world workflow applications and the results demonstrate that the proposed algorithm outperforms other state-of-the-art algorithms.

Cost-Effective Resource Provisioning for Real-Time Workflow in Cloud

In the era of big data, mining and analysis of the enormous amount of data has been widely used to support decision-making. This complex process including huge-volume data collecting, storage, transmission, and analysis could be modeled as workflow. Meanwhile, cloud environment provides sufficient computing and storage resources for big data management and analytics. Due to the clouds providing the pay-as-you-go pricing scheme, executing a workflow in clouds should pay for the provisioned resources. Thus, cost-effective resource provisioning for workflow in clouds is still a critical challenge. Also, the responses of the complex data management process are usually required to be real-time. Therefore, deadline is the most crucial constraint for workflow execution. In order to address the challenge of cost-effective resource provisioning while meeting the real-time requirements of workflow execution, a resource provisioning strategy based on dynamic programming is proposed to achieve cost-effectiveness of workflow execution in clouds and a critical-path based workflow partition algorithm is presented to guarantee that the workflow can be completed before deadline. Our approach is evaluated by simulation experiments with real-time workflows of different sizes and different structures. The results demonstrate that our algorithm outperforms the existing classical algorithms.

An Optimal Time-Based Resource Allocation for Biomedical Workflow Applications in Cloud

ABSTRACT Biomedical workflow applications’ necessities have increased with the progress of the biotechnology industry and it involves large volumes of data which require large-scale computation. Due to the upcoming data deluge of biomedical data, processing and scheduling of the same in computing resources have presented many challenges. A biomedical workflow application requires heterogeneous resources for execution. Thus, it utilizes the advantage of cloud computing, as a cloud provides scalable high-performance computing resources. Scheduling of these workflows in the cloud is a crucial problem for which the proposed solution contributes greatly towards execution improvement. The proposed algorithm provides optimal and effective utilization of computational resources by novel scheduling of algorithms. Least Execution Time Cloud Workflow Scheduling (LETCWS) and Revised Heterogeneous Earliest Finish Time (RHEFT) algorithms are proposed and implemented in two distinct levels to schedule the biomedical workflow applications in such a way that it reduces the execution time while reducing the cost. The proposed algorithms are evaluated using the Workflowsim tool with real-world biomedical workflow applications. Experimental results illustrate the performance of scheduling workflow applications over the other existing approaches. The result shows that the proposed algorithm achieves better performance in terms of execution time and cost.

Protecting scientific workflows in clouds with an intrusion tolerant system

With the development of cloud computing technology, more and more scientific workflows are delivered to cloud platforms to complete. However, there are many threats in clouds due to the multi-tenant coexistence. In order to protect scientific workflows in clouds, the authors propose an intrusion tolerant scientific workflow system. In this system, the task executors containing multiple virtual machines are used for workflow sub-task execution to enhance reliability. Then lagged decision mechanism is presented to ensure uninterrupted workflow execution while checking the intermediate data, and assessing the confidence of these data. Inspired by moving target defence, they propose a dynamic task scheduling strategy based on resource circulation to periodically generate and recycle task executors, keeping the clean state of the workflow execution environment. Furthermore, temporary workflow intermediate data backup mechanism is presented, the stored intermediate data can be used for the re-execution of workflow sub-tasks with low confidence. Experiments are conducted in both the actual test environment based on OpenStack and the simulated test environment based on WorkflowSim toolkit. Experimental results demonstrate that the proposed system can effectively enhance intrusion tolerance of scientific workflows.

Big Data Processing Workflows Oriented Real-Time Scheduling Algorithm using Task-Duplication in Geo-Distributed Clouds

Scheduling big data processing workflows involves both large-scale tasks and transmission of massive intermediate data among tasks, thus optimizing their completion time and monetary cost becomes a challenging issue. Besides, data streams are continuously generated, and dynamically submitted to clouds for real-time or near real-time processing. Naturally, responsive schedules are required to keep pace with such dynamic environments and this further aggravates the difficulty of the workflow scheduling problem. To address these issues, we first derive two theorems to minimize the completion time of a set of parallel workflow tasks and the start time of each workflow task, and then define the latest finish time for workflow tasks, which is also proved its advantage in reducing costs without delaying the completion of workflows. On the basis of these theorems, we propose a novel r eal- t ime s cheduling a lgorithm using t ask- d uplication, RTSATD, such that minimizing both the completion time and monetary cost of processing big data workflows in clouds. The performance of RTSATD is analyzed by using both synthesized and real-world workflows. The experimental results demonstrate the superiority of the proposed algorithm with respect to completion time (up to 28.73 percent) and resource utilization (up to 46.31 percent) over two existing approaches.

Scheduling Workflows With Composite Tasks: A Nested Particle Swarm Optimization Approach

Scientific cloud workflows enable the access to distributed computing resources in cloud environments for executing scientific computing applications. In the literature, most workflow scheduling models assume that each workflow task is mapped to only one service instance. But in computation and data-intensive applications, it is common that the computation resources provided by a single service instance are insufficient for some complicated tasks which contain several closely correlated sub-tasks. To manage such complicated workflows, this article devises a novel workflow model with composite tasks (cWFS). The model views a complicated task as a composite task and allows mapping multiple service instances to a composite task. The data transmission among sub-tasks of a composite task can also be addressed by the proposed model. To solve cWFS problem, we devise a nested particle swarm optimization (N-PSO) that utilizes two kinds of populations, i.e., the outer population and inner population. Since N-PSO is a bit time-consuming, we further devise a Fast version of N-PSO (FN-PSO), which can save more than 60 percent of running time compared with N-PSO. The proposed approaches are evaluated on five real-world workflow types. The experimental results verify that the proposed approaches can solve the new workflow model effectively.

Deep‐Q learning‐based heterogeneous earliest finish time scheduling algorithm for scientific workflows in cloud

SummaryThe complex and large‐scale scientific workflow applications are effectively executes on the cloud. The performance of cloud computing highly depends on the task scheduling. Optimal workflow scheduling is still a challenge that needs to be addressed due to the conflicting objectives and increasing demand for quality of service. Task scheduling is an NP‐hard problem due to its complexity. The newly introduced methods for resolving the problem of task scheduling are facing challenges to take the benefits of all aspects of cloud computing. In this article, we study the joint optimization of cost and makespan of scheduling workflows in infrastructure as a service clouds and propose a new workflow scheduling scheme using deep learning. In this scheme, a deep‐Q learning‐based heterogeneous earliest‐finish‐time (DQ‐HEFT) algorithm is developed, which closely integrates the deep learning mechanism with the task scheduling heuristic HEFT. The workflowsim simulator is used for the experiment of the real‐world and synthetic workflows. The experiment results demonstrate the efficiency of our proposed approach compared with existing algorithms. This technique can achieve significantly better makespan and speed metrics with a remarkably higher volume of data and can run faster compared with the existing workflow scheduling algorithms in cloud computing environment.

Energy-Aware Cloud Workflow Applications Scheduling With Geo-Distributed Data

Electricity prices differ during different time periods and change from place to place. Cloud workflow applications often require geo-distributed data which is transmitted among heterogeneous servers in intra- and inter- data centers. Such varying electricity prices and data transmission time bring great challenges when optimizing the energy cost for scheduling tasks in workflow applications to heterogeneous servers in cloud data centers. In this article, we minimize the total electricity cost in a deadline constrained energy-aware workflow scheduling problem with data being geographically distributed across data centers. A scheduling algorithm is proposed. Strategies are developed to sequence workflow applications, divide deadlines and sort tasks. An adaptive local search method is presented to improve solutions during the search process which dynamically balances intensification using neighborhood structures of increasing size. Components and parameter values are statistically calibrated over a comprehensive set of random instances. The proposed algorithm is compared to modified classical algorithms for similar problems. Experimental results demonstrate the effectiveness of the proposal for the considered problem.

Design of Energy Aware Scheduling Algorithm for Executing Scientific Workflows in Cloud

The usage of cloud computing and its resources for the execution of scientific workflow is a rapidly increasing demand. The Scientific applications are generally large in scale; even a single scientific workflow includes more number of complex tasks. Execution of these tasks can be made successful only by deploying it in the cloud virtual machines, because only cloud environment can only provide very large number of computing assets. In cloud, every processing resource is given as Virtual Machine. Any scientific workflow deployed in the cloud needs large number of virtual machines so; huge amount of computational energy is spent by the virtual machines to execute multifaceted scientific workflows. Hence there arises the need to utilize the cloud resources in an energy efficient way. Also, if the virtual machines are planned to schedule in an energy efficient manner there is an increase of makepsan of the workflow which is going to be an important parameter for completing the workflow within the deadline. So, the need for executing scientific workflows in energy efficient way with reduced makespan becomes a major issue among the researchers. It also becomes very challenging task to executing a scientific workflow in within the given deadline of a task in the given workflow. To address these issues, a new Energy Aware workflow scheduling algorithm is proposed and designed with improved makespan for the execution of different scientific applications in cloud environment.

Data-Oriented Scheduling with Dynamic-Clustering Fault-Tolerant Technique for Scientific Workflows in Clouds

Cloud computing is one of the most prominent parallel and distributed computing paradigm. It is used for providing solution to a huge number of scientific and business applications. Large scale scientific applications which are structured as scientific workflows are evaluated through cloud computing. Scientific workflows are data-intensive applications, as a single scientific workflow may consist of hundred thousands of tasks. Task failures, deadline constraints, budget constraints and improper management of tasks can also instigate inconvenience. Therefore, provision of fault-tolerant techniques with data-oriented scheduling is an important approach for execution of scientific workflows in Cloud computing. Accordingly, we have presented enhanced data-oriented scheduling with Dynamic-clustering fault-tolerant technique (EDS-DC) for execution of scientific workflows in cloud computing. We have presented data-oriented scheduling as a proposed scheduling technique. We have also equipped EDS-DC with Dynamic-clustering fault-tolerant technique. To know the effectiveness of EDS-DC, we compared its results with three well-known enhanced heuristic scheduling policies referred to as: (a) MCT-DC, (b) Max-min-DC, and (c) Min-min-DC. We considered scientific workflow of CyberShake as a case study, because it contains most of the characteristics of scientific workflows such as integration, disintegration, parallelism, and pipelining. The results show that EDS-DC reduced make-span of 10.9% as compared to MCT-DC, 13.7% as compared to Max-min-DC, and 6.4% as compared to Min-min-DC scheduling policies. Similarly, EDS-DC reduced the cost of 4% as compared to MCT-DC, 5.6% as compared to Max-min-DC, and 1.5% as compared to Min-min-DC scheduling policies. These results in respect of make-span and cost are highly significant for EDS-DC as compared with above referred three scheduling policies. The SLA is not violated for EDS-DC in respect of time and cost constraints, while it is violated number of times for MCT-DC, Max-min-DC, and Min-min-DC scheduling techniques.