Workflow Scheduling Research Articles

Scientific workflow scheduling algorithms in cloud environments: a comprehensive taxonomy, survey, and future directions

Scientific workflow scheduling algorithms in cloud environments: a comprehensive taxonomy, survey, and future directions

Workflow Scheduling in Cloud–Fog Computing Environments: A Systematic Literature Review

ABSTRACTThe Internet of Things (IoT) facilitates the connectivity of billions of physical devices for exchanging information and enabling a wide range of applications. These applications can be presented in the form of dependent tasks, as outlined in a workflow. These workflows face limitations due to constraints in IoT sensors. To address these limitations, cloud computing has emerged to offer a large capacity of computing and storing with a great capability to adjust resources according to the need. However, cloud computing might not adequately meet the low‐latency of IoT workflow requirements when scheduling a workflow composed of IoT tasks due to its centralized nature. Moreover, cloud computing is not ideal for delay‐sensitive workflows and may increase communication costs. In response to these challenges, the use of fog computing as an extension to cloud computing scheme is recommended. Fog computing aims to process workflow tasks close to IoT devices. While fog computing offers various advantages, integrating these systems into workflow scheduling remains one of the most formidable challenges in distributed environments. Indeed, significant issues arise due to the timely execution and the resource limitations. In this survey paper, we present a Systematic Literature Review (SLR) on the current state of the art in this domain. We propose a taxonomy to compare and evaluate the existing studies on workflow scheduling approaches in cloud–fog computing environments. This taxonomy encompasses various criteria, including scheduling techniques, performance metrics, workflow dependencies, scheduling policies, and evaluation tools. We highlight certain recommendations for open issues which require more investigations. Our aim is to provide valuable insights for researchers and developers interested in understanding the contributions and challenges of current workflow scheduling approaches in cloud–fog computing environments.

A tri-chromosome-based evolutionary algorithm for energy-efficient workflow scheduling in clouds

Cloud computing is increasingly attracting workflow applications, where workflows need to satisfy execution deadlines and energy consumption is to be minimized. So far, numerous studies have adopted evolutionary algorithms to optimize the energy consumption of workflow execution. Dynamic voltage and frequency scaling (DVFS) has been widely employed to save energy on computing devices running workflow tasks. However, most existing evolutionary algorithms focus on evolving task execution order or mapping from tasks to resources, while neglecting the evolution of task runtime to leverage the dynamic voltage and frequency scaling (DVFS) technology for further energy saving. To compensate for that deficiency, this paper designs a tri-chromosome-based evolutionary algorithm, namely TCEA, to evolve three types of decision vectors (i.e., task order, task and resource mapping, and task runtime) simultaneously using three problem-specific mechanisms. Firstly, we construct a search space by using the tasks’ minimum and optimal runtime, and propose a solution representation mechanism to simplify the decision vector for task runtime between 0 and 1. Secondly, we design a deadline constraint handling mechanism to distribute those durations exceeding the deadline to each task based on their extension of the minimum runtime. Thirdly, we exploit the workflow structure to cluster decision variables without direct constraints into the same group. During each iteration, only the order of tasks within a group evolves to avoid precedence constraints, thus performing searches within the feasible space. At last, we conduct comparison experiments on five types of real-world workflows with 30 to 1000 tasks. The energy consumed by TCEA is much less than those consumed by the state-of-the-art workflow scheduling algorithms, demonstrating the superior performance of TCEA in energy saving.

Enhancing workflow efficiency with a modified Firefly Algorithm for hybrid cloud edge environments

Efficient scheduling of scientific workflows in hybrid cloud-edge environments is crucial for optimizing resource utilization and minimizing completion time. In this study, we evaluate various scheduling algorithms, emphasizing the Modified Firefly Optimization Algorithm (ModFOA) and comparing it with established methods such as Ant Colony Optimization (ACO), Genetic Algorithm (GA), and Particle Swarm Optimization (PSO). We investigate key performance metrics, including makespan, resource utilization, and energy consumption, across both cloud and edge configurations. Scientific workflows often involve complex tasks with dependencies, which can challenge traditional scheduling algorithms. While existing methods show promise, they may not fully address the unique demands of hybrid cloud-edge environments, potentially leading to suboptimal outcomes. Our proposed ModFOA integrates cloud and edge computing resources, offering an effective solution for scheduling workflows in these hybrid environments. Through comparative analysis, ModFOA demonstrates improved performance in reducing makespan and completion times, while maintaining competitive resource utilization and energy efficiency. This study highlights the importance of incorporating cloud-edge integration in scheduling algorithms and showcases ModFOA’s potential to enhance workflow efficiency and resource management across hybrid environments. Future research should focus on refining ModFOA’s parameters and validating its effectiveness in practical hybrid cloud-edge scenarios.

Reinforcement Learning-Based Online Scheduling of Multiple Workflows in Edge Environment

Reinforcement Learning-Based Online Scheduling of Multiple Workflows in Edge Environment

Multi-Objective Reinforcement Learning Based Algorithm for Dynamic Workflow Scheduling in Cloud Computing

Multi-Objective Reinforcement Learning Based Algorithm for Dynamic Workflow Scheduling in Cloud Computing

Retraction Note: A modified shuffled frog leaping algorithm for scientific workflow scheduling using clustering techniques

Retraction Note: A modified shuffled frog leaping algorithm for scientific workflow scheduling using clustering techniques

C-KHCS: Multi-Objective Workflow Scheduling using Chaotic Krill Herd Optimization and Improved Cuckoo Search in Fog Computing

C-KHCS: Multi-Objective Workflow Scheduling using Chaotic Krill Herd Optimization and Improved Cuckoo Search in Fog Computing

SpotDAG: An RL-Based Algorithm for DAG Workflow Scheduling in Heterogeneous Cloud Environments

SpotDAG: An RL-Based Algorithm for DAG Workflow Scheduling in Heterogeneous Cloud Environments

Workflow scheduling based on asynchronous advantage actor–critic algorithm in multi-cloud environment

Recently, the multi-cloud environment (MCE) has increasingly become the preferred choice of users. As with the cloud environment, efficient workflow scheduling in a MCE remains crucial for identifying the cost efficiency and overall performance of the MCE. In MCE, the resources exhibit heterogeneity, complexity, and dynamism. Simultaneously, the intricate inter-task dependencies among workflow tasks, diverse Quality of Service (QoS) metrics for users, and multiple cloud service providers’ (CSPs) billing mechanisms significantly amplify the workflow scheduling challenge. Motivated by the application of reinforcement learning (RL) in workflow scheduling in a cloud environment, this paper proposes a scheduling algorithm that takes advantage of the asynchronous advantage actor–critic algorithm (A3C) to balance cost, makespan and resource utilization in workflow scheduling in a MCE. By analyzing the elements in the MCE, we design and define multiple agents in the MCE, and each cloud service provider will have an agent to record the state and update the local parameters. For the workflow task submitted by the user, the action is selected according to the initialization policy and submitted to the scheduling action to allocate the task to a designated virtual machine in the MCE so that each agent can more clearly perceive the environment change and adapt to the MCE. In contrast to the traditional A3C algorithm, we design a new critic network according to the data characteristics of real-world scientific workflows so that each agent is more suitable for real-world scientific workflow data. Through multiple sets of simulation experiments, the workflow scheduling algorithm based on the A3C algorithm in the MCE (MCWS-A3C) was compared with three benchmark methods. The experimental results show that the proposed method has better advantages than other methods in terms of cost, makespan, and resource utilization. Specifically, on the Montage_100 dataset, the average cost was reduced by 55.12% compared to other methods. The pioneering introduction of the A3C algorithm that adapts to the dynamic environment into the MCE brings more possibilities to address the issue of workflow scheduling in the MCE.

An electricity price and energy-efficient workflow scheduling in geographically distributed cloud data centers

The cloud computing platform has become a favorable destination for running cloud workflow applications. However, they are primarily complicated and require intensive computing. Task scheduling in cloud environments, when formulated as an optimization problem, is proven to be NP-hard. Thus, efficient task scheduling plays a decisive role in minimizing energy costs. Electricity prices fluctuate depending on the vending company, time, and location. Therefore, optimizing energy costs has become a serious issue that one must consider when building workflow applications scheduling across geographically distributed cloud data centers (GD-CDCs). To tackle this issue, we have suggested a dual optimization approach called electricity price and energy-efficient (EPEE) workflow scheduling algorithm that simultaneously considers energy efficiency and fluctuating electricity prices across GD-CDCs, aims to reach the minimum electricity costs of workflow applications under the deadline constraints. This novel integration of dynamic voltage and frequency scaling (DVFS) with energy and electricity price optimization is unique compared to existing methods. Moreover, our EPEE approach, which includes task prioritization, deadline partitioning, data center selection based on energy efficiency and price diversity, and dynamic task scheduling, provides a comprehensive solution that significantly reduces electricity costs and enhances resource utilization. In addition, the inclusion of both generated and original data transmission times further differentiates our approach, offering a more realistic and practical solution for cloud service providers (CSPs). The experimental results reveal that the EPEE model produces better success rates to meet task deadlines, maximize resource utilization, cost and energy efficiencies in comparison to adapted state-of-the-art algorithms for similar problems.

Secure workflow scheduling algorithm utilizing hybrid optimization in mobile edge computing environments

The rapid advancement of mobile communication technology and devices has greatly improved our way of life. It also presents a new possibility that data sources can be used to accomplish computing tasks at nearby locations. Mobile Edge Computing (MEC) is a computing model that provides computer resources specifically designed to handle mobile tasks. Nevertheless, there are certain obstacles that must be carefully tackled, specifically regarding the security and quality of services in the workflow scheduling over MEC. This research proposes a new method called Feedback Artificial Remora Optimization (FARO)-based workflow scheduling method to address the issues of scheduling processes with improved security in MEC. In this context, the fitness functions that are taken into account include multi-objective, such as CPU utilization, memory utilization, encryption cost, and execution time. These functions are used to enhance the scheduling of workflow tasks based on security considerations. The FARO algorithm is a combination of the Feedback Artificial Tree (FAT) and the Remora Optimization Algorithm (ROA). The experimental findings have demonstrated that the developed approach surpassed current methods by a large margin in terms of CPU use, memory consumption, encryption cost, and execution time, with values of 0.012, 0.010, 0.017, and 0.036, respectively.

Clustering-based multi-objective optimization considering fairness for multi-workflow scheduling on clouds

Distributed computing, such as cloud computing, provides promising platforms for orchestrating scientific workflows' tasks based on their sequences and dependencies. Workflow scheduling plays an important role in optimizing concerned objectives for distributed computing, such as minimizing the makespan and cost. Many researchers have focused on optimizing a specific single workflow with multiple objectives. Currently, there are few studies on multi-workflow scheduling, with most research focusing on objectives such as cost and makespan. However, multi-workflow scheduling requires the design of specific objectives that reflect the unique characteristics of multiple workflows. On the other hand, clustering-based approaches have garnered significant attention in the field of workflow scheduling over distributed computing resources due to their advantage in reducing data communication among tasks. Despite this, the effectiveness of clustering-based algorithms has not been extensively studied and validated in the context of multi-objective multi-workflow scheduling models. Motivated by these factors, we propose an approach for multiple workflows' multi-objective optimization (MOO), considering the new defined metric, fairness. We first mathematically formulate the fairness and define a fairness-involved MOO model. Then, we propose an advanced clustering-based resource optimization strategy in multiple workflow runs. Experimental results show that the proposed approach performs better than the compared algorithms without significant compromise of the overall makespan and cost as well as individual fairness, which can guide the simulation workflow scheduling on clouds.

Dynamic deadline constrained multi-objective workflow scheduling in multi-cloud environments

Workflow scheduling becomes difficult and demanding in multi-cloud systems because of the variety of billing models and resource kinds, as well as the susceptibility of processes to time limitations. In which the execution duration of the workflow needs to be flexibly adjusted according to the urgency or otherwise of the real situation, in this work, we model the workflow scheduling problem with deadline constraints as a dynamically constrained multi-objective optimization problem (DCMOP), where the disruption rate of the special cloud resources and the failure probability in different cloud environments are expressed as the disruption probability of the workflow in a comprehensive manner while considering the execution time and cost. Dynamics arise from price changes for specialized cloud resources and changes in the execution duration of workflows. In addition, an algorithm for dynamically constrained multi-objective optimization (DC-MOEA-DPDS) is proposed in this paper. The algorithm based on two-population synergy as well as diversity selection. The auxiliary population ignores constraint limitations to help the main population speed up convergence, and diversity selection enables the auxiliary population to have better diversity to assist the general public in looking for more viable areas. Through dynamic constrained workflow simulation experiments in a multi-cloud environment, our algorithm reduces the execution time by an average of 13.29%, the cost by an average of 32.26%, and the interruption probability by 56.75%. In addition, our algorithm outperforms other algorithms in experiments on a dynamically constrained benchmark set.

A cooperative coevolutionary genetic programming hyper-heuristic for multi-objective makespan and cost optimization in cloud workflow scheduling

This study presents a novel multi-objective approach for NP-hard workflow scheduling in cloud computing environments. Traditional rule-based heuristics offer flexibility but lack consistent superiority across all criteria and scenarios. The development of specific rules usually requires tedious manual refinement by experts. To overcome this limitation, our method leverages evolutionary computation and simulation to automatically derive new rules. Moreover, workflow scheduling involves two crucial and related aspects: task scheduling and the allocation of virtual machines. Our contribution includes three distinct algorithms: two that address each decision separately and a third approach that coevolves priority rules for both decisions simultaneously. Computational tests demonstrate superior performance, with an exploration of rules yielding a 72.91% larger hypervolume for optimizing makespan and costs compared to benchmark heuristics from the literature. The validation on unseen instances shows a 90.26% improvement in hypervolume performance, highlighting the robustness of our approach.

Optimizing scientific workflow scheduling in cloud computing: a multi-level approach using whale optimization algorithm

Cloud computing has evolved into an indispensable tool for facilitating scientific research due to its ability to efficiently distribute and process workloads in a virtual environment. Scientific tasks that involve complicated task dependencies and user-defined constraints related to quality of service (QoS) and time constraints require the efficient use of cloud resources. Planning these scientific workflow tasks represents an NP-complete problem, prompting researchers to explore various solutions, including conventional planners and evolutionary optimization algorithms. In this study, we present a novel, multistage algorithm specifically designed to schedule scientific workflows in cloud computing contexts. This approach addresses the challenges of efficiently mapping complex workflows onto distributed cloud resources while considering factors like resource heterogeneity, dynamic workloads, and stringent performance requirements. The algorithm uses the whale optimization algorithm (WOA) with a two-phase approach to shorten execution time, minimize financial costs, and effectively maintain load balancing.

Machine learning‐driven implementation of workflow optimization in cloud computing for IoT applications

AbstractThe optimization of workflow scheduling in Internet of Things (IoT) environments presents significant challenges due to the dynamic and heterogeneous nature of these systems. Traditional techniques must often adapt to fluctuating network conditions and varying data loads. To address these limitations, we propose a novel approach that leverages Automated Machine Learning (AutoML) integrated with cloud computing to optimize workflow scheduling for IoT applications. Our solution automates machine learning model selection, training, and tuning, significantly enhancing computational efficiency and adaptability. Through extensive experimentation, we demonstrate that our AutoML‐driven approach surpasses conventional algorithms across several key metrics, including accuracy, computational efficiency, adaptability to dynamic environments, and communication efficiency. Specifically, our method achieves a scheduling accuracy improvement of up to 25%, a reduced computational overhead by 30%, and a 40% enhancement in adaptability under dynamic conditions. Furthermore, the scalability of our solution is critical in cloud computing contexts, enabling efficient handling of large‐scale IoT deployments by leveraging cloud resources for distributed processing. This scalability ensures that our approach can effectively manage increasing data volumes and device heterogeneity inherent in modern IoT systems.

Deadline-constrained security-aware workflow scheduling in hybrid cloud architecture

A hybrid cloud is an efficient solution to deal with the problem of insufficient resources of a private cloud when computing demands increase beyond its resource capacities. Cost-efficient workflow scheduling, considering security requirements and data dependency among tasks, is a prominent issue in the hybrid cloud. To address this problem, we propose a mathematical model that minimizes the monetary cost of executing a workflow and satisfies the security requirements of tasks under a deadline. The proposed model fulfills data dependency among tasks, and data transmission time is formulated with exact mathematical expressions. The derived model is a Mixed-integer linear programming problem. We evaluate the proposed model with real-world workflows over changes in the input variables of the model, such as the deadline and security requirements. This paper also presents a post-optimality analysis that investigates the stability of the assignment problem. The experimental results show that the proposed model minimizes the cost by decreasing inter-cloud communications for dependent tasks. However, the optimal solutions are affected by the limitations that are imposed by the problem constraints.

A cost, time, energy-aware workflow scheduling using adaptive PSO algorithm in a cloud–fog environment

A cost, time, energy-aware workflow scheduling using adaptive PSO algorithm in a cloud–fog environment

A deep reinforcement learning-based scheduling framework for real-time workflows in the cloud environment

Benefiting from the flexibility, scalability and many other advantages of cloud computing, an increasing number of research institutions and enterprises are using cloud platforms to deploy their scientific or commercial workflow applications. From the perspective of cloud service providers, how to schedule workflows reasonably to improve user satisfaction and resource utilization is a big challenge. This challenge becomes more prominent in a dynamic cloud environment, where the workload of user-submitted workflows is constantly changing and unpredictable. Traditional scheduling methods designed for fixed workflows or real-time independent tasks are not applicable in such scenarios. In this paper, we present an online scheduling framework for multiple real-time workflows in a dynamic environment. In this framework, the scheduler employs a deep reinforcement learning based algorithm (R-DQN) to assign each task in workflows to a suitable virtual machine instance according to the current situation. Its goal is to achieve optimal task scheduling and resource allocation, so as to minimize workflow makespan and maximize resource utilization. We provide a comprehensive overview of the design and implementation of our approach, and we conduct experiments using various workflow benchmarks with different structures and scales on the WorkflowSim platform. The experimental results demonstrate that our algorithm significantly outperforms the compared algorithms in term of makespan and other metrics. Moreover, our algorithm demonstrates strong performance in generality, robustness and scalability.