Dynamic Workflow Scheduling Research Articles

Multi-Objective Reinforcement Learning Based Algorithm for Dynamic Workflow Scheduling in Cloud Computing

Multi-Objective Reinforcement Learning Based Algorithm for Dynamic Workflow Scheduling in Cloud Computing

Enhancing generalization in genetic programming hyper-heuristics through mini-batch sampling strategies for dynamic workflow scheduling

Genetic Programming Hyper-heuristics (GPHH) have been successfully used to evolve scheduling rules for Dynamic Workflow Scheduling (DWS) as well as other challenging combinatorial optimization problems. The method of sampling training instances has a significant impact on the generalization ability of GPHH, yet they are rarely addressed in existing research. This article aims to fill this gap by proposing a GPHH algorithm with a sampling strategy to thoroughly investigate the impact of six instance sampling strategies on algorithmic generalization, including one rotation strategy, three mini-batch strategies, and two hybrid strategies. Experiments across four scenarios with varying settings reveal that: (1) mini-batch with random sampling can outperform rotation in generalizing to unseen workflow scheduling problems under the same computational cost; (2) employing a hybrid strategy that combines rotation and mini-batch further enhances the generalization ability of GPHH; and (3) mini-batch and hybrid strategies can effectively enable heuristics trained on small-scale training instances generalizing well to large-scale unseen ones. These findings highlight the potential of mini-batch strategies in GPHH, offering improved generalization performance while maintaining diversity and suggesting promising avenues for further exploration in GPHH domains.

Dual-Tree Genetic Programming With Adaptive Mutation for Dynamic Workflow Scheduling in Cloud Computing

Dual-Tree Genetic Programming With Adaptive Mutation for Dynamic Workflow Scheduling in Cloud Computing

Transfer Learning Based Multi-Objective Evolutionary Algorithm for Dynamic Workflow Scheduling in the Cloud

Transfer Learning Based Multi-Objective Evolutionary Algorithm for Dynamic Workflow Scheduling in the Cloud

Dynamic multi-objective workflow scheduling for combined resources in cloud

Cloud resource providers offer idle resources to users as spot instances. The price of the instances changes with market supply and demand, and the dynamic price can have a significant impact on workflow scheduling. In this work, we use a combination of spot and on-demand instances as the foundation cloud resource and characterize the dynamic workflow scheduling problem as a dynamic multi-objective optimization problem (DMOP), where the dynamics originate from the dynamic price of spot instances. The scheduling solution is found by considering three objectives: maximizing the reliability of the instances while minimizing the makespan and cost. In addition, we provide an enhanced MOEAD algorithm called MOEA/D-URDI that combines diversity introduction and uniform random sampling, where the uniform random sampling paradigm is used to generate the initial weight vector. The dynamic multi-objective optimization evolutionary algorithm DMOEA/D-URDI is then created by combining the method with a dynamic optimization framework. Our technique beats existing algorithms, according to experimental data based on dynamic benchmark sets and three well-known scientific procedures in terms of metrics on dynamic benchmark sets and better ensures reliability in scheduling scientific workflows while reducing makespan and cost.

Integrating Cognition Cost With Reliability QoS for Dynamic Workflow Scheduling Using Reinforcement Learning

The rapid rise of microservice architecture poses severe challenges to workflow scheduling, resource allocation, and goal optimization. However, faults and failures usually happen during workflow running. To ensure the workflow's successful execution during scheduling microservices, this paper proposes a dynamic workflow scheduling algorithm by integrating cognition cost and reliability QoS for using reinforcement learning (WS-CCR). Firstly, we explore the ‘restart policy’ of containers in the Kubernetes architecture, which lays the foundation for our work that adopts the redundancy strategy to ensure workflow operation. Then we consider the cognitive cost based on the fact that users have a cognitive process for different microservices in selecting microservices. Additionally, another optimization goal is the reliability of workflows. On this basis, we design a reasonable reward function in reinforcement learning to generate dynamic strategies. Furthermore, following some generated strategies, our engine will schedule candidate microservices for tasks to execute step by step. A series of experiments on Alibaba and business areas workflows have proven the superior performance of our algorithm. Our WS-CCR can generate better Pareto solution sets than other baselines in terms of improving the reliability of running workflows. Finally, we give a case study to prove the practicability of our method.

Genetic Programming for Dynamic Workflow Scheduling in Fog Computing

<bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</b> ynamic <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">W</b> orkflow <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S</b> cheduling in <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</b> og <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</b> omputing (DWSFC) is an important optimisation problem with many real-world applications. The current workflow scheduling problems only consider cloud servers but ignore the roles of mobile devices and edge servers. Some applications need to consider the mobile devices, edge, and cloud servers simultaneously, making them work together to generate an effective schedule. In this article, a new problem model for DWSFC is considered and a new simulator is designed for the new DWSFC problem model. The designed simulator takes the mobile devices, edge, and cloud servers as a whole system, where they all can execute tasks. In the designed simulator, two kinds of decision points are considered, which are the routing decision points and the sequencing decision points. To solve this problem, a new <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">M</b> ulti- <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">T</b> ree <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">G</b> enetic <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">P</b> rogramming (MTGP) method is developed to automatically evolve scheduling heuristics that can make effective real-time decisions on these decision points. The proposed MTGP method with a multi-tree representation can handle the routing decision points and sequencing decision points simultaneously. The experimental results show that the proposed MTGP can achieve significantly better test performance (reduce the makespan by up to 50%) on all the tested scenarios than existing state-of-the-art methods.

An MCDM optimization based dynamic workflow scheduling used to handle priority tasks for fault tolerance in IIOT

Cloud computing's prominence can be mainly attributed to its convenient on-demand service delivery via the internet. The difficulty in managing these requests and allocating them optimally is exacerbated by the growing number of users making them and the variety of criteria used to apply cloud resources. Some of these difficulties include scheduling work while considering the user's Quality of Service (QoS) preferences, ensuring adequate load balancing, and adapting to the ever-changing conditions of the cloud environment. As server capabilities evolve, static load balancers, which distribute requests using fixed ratios of available resources, fail to provide optimum throughput. It has been shown that no other scheduling algorithms offer an adaptive method that considers both load balancing and maximizing the QoS criteria. The authors of this study present a novel adaptive strategy for dealing with this issue by fusing the best-worst multi-criteria decision-making method (BWM) with the compromise ranking method (VIKOR). Task priorities are determined by using the VIKOR method as a decision-maker. Numerical experiments are used to verify the validity of the suggested method, and the methodology is compared to other scheduling methods in terms of a variety of performance criteria. The simulation findings show that the presented method outperforms its rivals regarding throughput, makespan, waiting time, virtual machine (VM) utilization, and VM usage cost across the board of simulated experimental settings.

A Multi-Objective Based Scheduling Framework for Effective Resource Utilization in Cloud Computing

Cloud computing is a promising platform for running massive workflow applications based on a pay-per-use model. In cloud computing, the reduction of energy consumption and providing security to workflow scheduling are the key research areas. The primary focus of the existing algorithms, viz., particle swarm optimization (PSO), crow Search optimization (CSO) and Other non-metaheuristic algorithms like Round Robbin(RR), SJF, Min-Min, Min-Max etc., is based on the execution time and cost of the workflow applications as a budget constraint. However, these algorithms failed to adequately determine energy consumption, resource utilization, and security in workflow scheduling. To address this issue, a multi-objective scheduling framework is proposed. In this paper, the framework performs dynamic workflow scheduling using universal unique identification- Blake (UUID-Blake), Manhattan Distance-Partition around algorithm (MD-PAM), Linear Scaling-Crow Search Optimization (LS-CSO), Anova-Recurrent Neural Network. The implementation of this framework was achieved in three phases (Phase 1, Phase 2, and Phase 3). Phase 1 is about user registration and authentication using UUID-Blake, which enhances security by allowing legitimate users into the cloud environment. Phase 2 deals with clustering and resource monitoring using MD-PAM and A-RNN, to reduce makespan the similar tasks are clustered using task length and maximize the resource utilization by predicting the resource availability. Phase 3 deals with the scheduling of dynamic workflows using LS-CSO by selecting suitable virtual machines. We have considered the heterogeneous computing scheduling problem (HCSP) and grid workload archive (GWA)-T-12 Bitbrains datasets for comparing our proposed framework with existing works. Based on the result analysis, the proposed LS-SCO outperformed when compared with the algorithms CSO,PSO and RR has achieved better performance.

Multihop Transmission-Oriented Dynamic Workflow Scheduling in Vehicular Cloud

Vehicular cloud as a kind of resource aggregation can provide cloud service flexibly for intelligent transportation systems and individual users. However, the dynamicity of vehicular cloud such as dynamic multihop transmission also brings new challenges to workflow scheduling. In this paper, a heuristic dynamic workflow scheduling (HDWS) strategy is proposed to solve the problem of workflow scheduling with dynamic multihop transmission in vehicular cloud. HDWS establishes a local scheduling based on the current resource status and task sets. The makespan of local scheduling is optimized by reassigning tasks based on task interdependency. At run time of an application, HDWS estimates the makespan in real time based on the current transmission rates and the unready tasks will be rescheduled once it detects the degradation of the makespan. Experimental results show that the proposed HDWS can improve the service success ratio of vehicular cloud and decrease the makespan compared to the existing approaches.

Cache-aware scheduling of scientific workflows in a multisite cloud

Many scientific experiments today are performed using scientific workflows, which become more and more data-intensive. We consider the efficient execution of such workflows in a multisite cloud, leveraging heterogeneous resources available at multiple geo-distributed data centers. Since it is common for workflow users to reuse code or data from previous workflows, a promising approach for efficient workflow execution is to cache intermediate data in order to avoid re-executing entire workflows. However, caching intermediate data and scheduling workflows to exploit such caching in a multisite cloud is complex. In particular, workflow scheduling must be cache-aware, in order to decide whether reusing cache data or re-executing workflows entirely. In this paper, we propose a solution for cache-aware scheduling of scientific workflows in a multisite cloud. Our solution includes a distributed and parallel architecture and new algorithms for adaptive caching, cache site selection, and dynamic workflow scheduling. We implemented our solution in the OpenAlea workflow system, together with cache-aware distributed scheduling algorithms. Our experimental evaluation in a three-site cloud with a real application in plant phenotyping shows that our solution can yield major performance gains, reducing total time up to 42% with 60% of the same input data for each new execution.

Dynamic workflow scheduling in the cloud using a neural network-based multi-objective evolutionary algorithm

Dynamic workflow scheduling in the cloud using a neural network-based multi-objective evolutionary algorithm

Dynamic Workflow Scheduling in Cloud using Neural Network based Multi-Objective Evolutionary Algorithm

Workflow is a series of jobs that are executed in order to complete a specific activity where the jobs are often dependent on each other. Data transfer that might occur between such jobs results into the creation of a workflow that aims at utilising resources for workflow tasks by optimising the use of cloud resources. Few of the existing single objectives workflows scheduling solutions have linearly combine multiple objectives to get a multi-objective solution, but it might not be able to model the real-world problem efficiently for certain conditions where the environment is dynamic in nature. Hence, a Neural Network based Multi-Objective Evolutionary Algorithm (NN-MOHEFT) that solves the multi-objective workflow scheduling issues in a dynamic environment was proposed in this article. The NN-MOHEFT learns the pattern behind changing Pareto optimal front for successive environment and tries to predict the Pareto optimal front for the next environment from the input Pareto optimal set of the current environment. The proposed NN-MOHEFT algorithm is at par with the original constructs when it comes to the hypervolume of objectives generated. It generates 10% more non-dominated solutions as compared to the original construct.

A Novel Approach to Scheduling Workflows Upon Cloud Resources with Fluctuating Performance

Cloud computing is recently getting increasingly popular for supporting scientific applications and complex business processes. Clouds are highly potent for executing workflow-based tasks due to the fact that they provide elastic resource provisioning styles through which computational-intensive workflows can obtain requested resources according to their elastic demand and establish execution environment over virtual machines (VMs). However, it remains a challenge to guarantee cost-effectiveness and quality of service of workflow deployed upon clouds due to the fact that real-world cloud infrastructures are usually with fluctuating and time-varying performance. Existing researches mainly consider that cloud infrastructures are with fixed, random, or bounded quality of service (QoS). In this work, however, we consider that scientific computing processes to be supported by decentralized cloud infrastructures with fluctuating QoS and aim at managing the monetary cost of workflows with the completion-time constraint to be satisfied. We address the performance-variation-aware workflow scheduling problem by leveraging a time-series-based prediction model and a Critical-Path-Duration-Estimation-based (CPDE for short) VM Selection strategy. The proposed method is capable of exploiting real-time trends of performance changes of cloud infrastructures and generating dynamic workflow scheduling plans. To prove the effectiveness of our proposed method, we perform extensive experimental case analysis over real-world third-party commercial clouds and show that our method clearly beats existing approaches.

Neural network based multi-objective evolutionary algorithm for dynamic workflow scheduling in cloud computing

Workflow scheduling is a largely studied research topic in cloud computing, which targets to utilize cloud resources for workflow tasks by considering the objectives specified in QoS. In this paper, we model dynamic workflow scheduling problem as a dynamic multi-objective optimization problem (DMOP) where the source of dynamism is based on both resource failures and the number of objectives which may change over time. Software faults and/or hardware faults may cause the first type of dynamism. On the other hand, confronting real-life scenarios in cloud computing may change number of objectives at runtime during the execution of a workflow. In this study, we propose a prediction-based dynamic multi-objective evolutionary algorithm, called NN-DNSGA-II algorithm, by incorporating artificial neural network with the NSGA-II algorithm. Additionally, five leading non-prediction based dynamic algorithms from the literature are adapted for the dynamic workflow scheduling problem. Scheduling solutions are found by the consideration of six objectives: minimization of makespan, cost, energy and degree of imbalance; and maximization of reliability and utilization. The empirical study based on real-world applications from Pegasus workflow management system reveals that our NN-DNSGA-II algorithm significantly outperforms the other alternatives in most cases with respect to metrics used for DMOPs with unknown true Pareto-optimal front, including the number of non-dominated solutions, Schott’s spacing and Hypervolume indicator.

A Truthful Dynamic Workflow Scheduling Mechanism for Commercial Multicloud Environments

The ultimate goal of cloud providers by providing resources is increasing their revenues. This goal leads to a selfish behavior that negatively affects the users of a commercial multicloud environment. In this paper, we introduce a pricing model and a truthful mechanism for scheduling single tasks considering two objectives: monetary cost and completion time. With respect to the social cost of the mechanism, i.e., minimizing the completion time and monetary cost, we extend the mechanism for dynamic scheduling of scientific workflows. We theoretically analyze the truthfulness and the efficiency of the mechanism and present extensive experimental results showing significant impact of the selfish behavior of the cloud providers on the efficiency of the whole system. The experiments conducted using real-world and synthetic workflow applications demonstrate that our solutions dominate in most cases the Pareto-optimal solutions estimated by two classical multiobjective evolutionary algorithms.

Investigating the robustness of re-scheduling policies with multi-agent system simulation

Increased complexity of current manufacturing systems together with dynamic conditions and permanent demands for flexible and robust functionality makes their management and control very difficult and challenging. Workflow simulation is an effective approach to investigate dynamic workflow scheduling policies and evaluate the overall manufacturing system performance. The results attained in simulation model can give directions on how to maximize system output when selecting an appropriate scheduling practice for a real system. In this paper, we investigate the abilities of multi-agent systems in combination with dynamic dispatching rules and failure handling mechanisms to manage dynamic environment conditions (such as machine failures) for systems in the production automation domain. We measure system robustness by systematically assessing the total system performance (e.g., number of finished products) in a number of representative test cases. We use an agent-based simulation environment, MAST, which has been validated with real-world hardware to strengthen the external validity of the simulation results. We investigated the performance of a re-scheduling component which uses four different policies that define how to adjust the system schedule in case of machine disturbances/failures. In the context of the empirical study the Complete Rerouting re-scheduling policy outperformed all other policies.