Execution Of Scientific Workflows Research Articles

Cost and performance aware scheduling technique for cloud computing environment

Recently, lot of interest have been put forth by researchers to improve workload scheduling in cloud platform. However, execution of scientific workflow on cloud platform is time consuming and expensive. As users are charged based on hour of usage, lot of research work have been emphasized in minimizing processing time for reduction of cost. However, the processing cost can be reduced by minimizing energy consumption especially when resources are heterogeneous in nature; very limited work have been done considering optimizing cost with energy and processing time parameters together in meeting task quality of service (QoS) requirement. This paper presents cost and performance aware workload scheduling (CPA-WS) technique under heterogeneous cloud platform. This paper presents a cost optimization model through minimization of processing time and energy dissipation for execution of task. Experiments are conducted using two widely used workflow such as Inspiral and CyberShake. The outcome shows the CPA-WS significantly reduces energy, time, and cost in comparison with standard workload scheduling model.

SWEL: A Domain-Specific Language for Modeling Data-Intensive Workflows

AbstractData-intensive applications aim at discovering valuable knowledge from large amounts of data coming from real-world sources. Typically, workflow languages are used to specify these applications, and their associated engines enable the execution of the specifications. However, as these applications become commonplace, new challenges arise. Existing workflow languages are normally platform-specific, which severely hinders their interoperability with other languages and execution engines. This also limits their reusability outside the platforms for which they were originally defined. Following the Design Science Research methodology, the paper presents SWEL (Scientific Workflow Execution Language). SWEL is a domain-specific modeling language for the specification of data-intensive workflows that follow the model-driven engineering principles, covering the high-level definition of tasks, information sources, platform requirements, and mappings to the target technologies. SWEL is platform-independent, enables collaboration among data scientists across multiple domains and facilitates interoperability. The evaluation results show that SWEL is suitable enough to represent the concepts and mechanisms of commonly used data-intensive workflows. Moreover, SWEL facilitates the development of related technologies such as editors, tools for exchanging knowledge assets between workflow management systems, and tools for collaborative workflow development.

Scientific workflow execution in the cloud using a dynamic runtime model

To explain specific phenomena, scientists perform a sequence of tasks, e.g., to gather, analyze and interpret data, forming a scientific workflow. Depending on the complexity of the workflow, scientists require access to various kinds of tools, applications and infrastructures for individual tasks. Current approaches are often limited to managing these resources at design time, requiring the scientist to preemptively set up applications essential for their workflow. Therefore, a dynamic provisioning and configuration of computing resources are required that fulfills these needs at runtime. In this paper, we present a dynamic runtime model that couples workflow tasks with their individual applications and infrastructure requirements. This runtime model is used as a knowledge base by a model-driven workflow execution engine orchestrating the sequence of tasks and their infrastructure. We exhibit that the simplicity of the runtime model supports the creation of highly tailored infrastructures, the integration of self-developed applications, as well as a human-in-the-loop allowing scientists to monitor and interact with the workflow at runtime. To tackle the heterogeneity of cloud provider interfaces, we implement the workflow runtime model by extending the Open Cloud Computing Interface cloud standard, which provides an extensible data model as well as a uniform interface to manage cloud resources. We demonstrate the applicability of our approach using three case studies and discuss the benefits of the runtime model from a user and system perspective.

Graph neural networks for detecting anomalies in scientific workflows

Identifying and addressing anomalies in complex, distributed systems can be challenging for reliable execution of scientific workflows. We model these workflows as directed acyclic graphs (DAGs), where the nodes and edges of the DAGs represent jobs and their dependencies, respectively. We develop graph neural networks (GNNs) to learn patterns in the DAGs and to detect anomalies at the node (job) and graph (workflow) levels. We investigate workflow-specific GNN models that are trained on a particular workflow and workflow-agnostic GNN models that are trained across the workflows. Our GNN models, which incorporate both individual job features and topological information from the workflow, show improved accuracy and efficiency compared to conventional learning methods for detecting anomalies. While joint trained with multiple scientific workflows, our GNN models reached an accuracy more than 80% for workflow level and 75% for job level anomalies. In addition, we illustrate the importance of hyperparameter tuning method in our study that can significantly improve the metric(s) measure of evaluating the GNN models. Finally, we integrate explainable GNN methods to provide insights on job features in the workflow that cause an anomaly.

Fault tolerance aware workload resource management technique for real‐time workload in heterogeneous computing environment

AbstractWorkload execution is composed by mapping larger tasks onto heterogeneous environments such as cloud platforms for enhancing the efficiency of workload resource management techniques. Execution of scientific workflow on a cloud platform is time‐consuming, expensive, and requires a fault‐tolerance guarantee. The existing methodology has emphasized on minimizing processing time to reduce costs. However, the processing cost can be reduced by minimizing energy consumption. In providing fault‐tolerance while meeting the workload quality of service requirement the task is offloaded to a new physical machine; Hence, this increases energy consumption and thereby increases the cost of workload execution. In addressing the research challenges this paper presents the fault‐tolerant aware (FTA) workload resource management (WRM) technique. First, the FTA‐WRM optimizes processing and communication costs as an energy constraint leveraging the dynamic voltage frequency scaling technique. Then, a task offloading mechanism is modeled as an energy constraint with an application delay requirement for providing fault tolerance into the FTA‐WRM. The experiment outcome shows the FTA‐WRM significantly improves processing and energy efficiency in comparison with the existing workload resource management technique.

A hybrid bi-objective scheduling algorithm for execution of scientific workflows on cloud platforms with execution time and reliability approach

A hybrid bi-objective scheduling algorithm for execution of scientific workflows on cloud platforms with execution time and reliability approach

Scientific Workflow Makespan Minimization in Edge Multiple Service Providers Environment

The edge computing model offers an ultimate platform to support scientific and real-time workflow-based applications over the edge of the network. However, scientific workflow scheduling and execution still facing challenges such as response time management and latency time. This leads to deal with the acquisition delay of servers, deployed at the edge of a network and reduces the overall completion time of workflow. Previous studies show that existing scheduling methods consider the static performance of the server and ignore the impact of resource acquisition delay when scheduling workflow tasks. Our proposed method presented a meta-heuristic algorithm to schedule the scientific workflow and minimize the overall completion time by properly managing the acquisition and transmission delays. We carry out extensive experiments and evaluations based on commercial clouds and various scientific workflow templates. The proposed method has approximately 7.7% better performance than the baseline algorithms, particularly in overall deadline constraint that gives a success rate.

Deadline-Constrained Cost-Effective Load-Balanced Improved Genetic Algorithm for Workflow Scheduling

Resource scheduling in a cloud computing environment is noteworthy for scientific workflow execution under a cost-effective deadline constraint. Although various researchers have proposed to resolve this critical issue by applying various meta-heuristic and heuristic approaches, no one is able to meet the strict deadline conditions with load-balanced among machines. This article has proposed an improved genetic algorithm that initializes the population with a greedy strategy. Greedy strategy assigns the task to a virtual machine that is under loaded instead of assigning the tasks randomly to a machine. In general workflow scheduling, task dependency is tested after each crossover and mutation operators of genetic algorithm, but here the authors perform after the mutation operation only which yield better results. The proposed model also considered booting time and performance variation of virtual machines. The authors compared the algorithm with previously developed heuristics and metaheuristics both and found it increases hit rate and load balance. It also reduces execution time and cost.

Real-time and dynamic fault-tolerant scheduling for scientific workflows in clouds

Cloud computing has become a popular technology for executing scientific workflows. However, with a large number of hosts and virtual machines (VMs) being deployed, the cloud resource failures, such as the permanent failure of hosts (HPF), the transient failure of hosts (HTF), and the transient failure of VMs (VMTF), bring the service reliability problem. Therefore, fault tolerance for time-consuming scientific workflows is highly essential in the cloud. However, existing fault-tolerant (FT) approaches consider only one or two above failure types and easily neglect the others, especially for the HTF. This paper proposes a Real-time and dynamic Fault-tolerant Scheduling (ReadyFS) algorithm for scientific workflow execution in a cloud, which guarantees deadline constraints and improves resource utilization even in the presence of any resource failure. Specifically, we first introduce two FT mechanisms, i.e., the replication with delay execution (RDE) and the checkpointing with delay execution (CDE), to cope with HPF and VMTF, simultaneously. Additionally, the rescheduling (ReSC) is devised to tackle the HTF that affects the resource availability of the entire cloud datacenter. Then, the resource adjustment (RA) strategy, including the resource scaling-up (RS-Up) and the resource scaling-down (RS-Down), is used to adjust resource demands and improve resource utilization dynamically. Finally, the ReadyFS algorithm is presented to schedule real-time scientific workflows by combining all the above FT mechanisms with RA strategy. We conduct the performance evaluation with real-world scientific workflows and compare ReadyFS with five vertical comparison algorithms and three horizontal comparison algorithms. Simulation results confirm that ReadyFS is indeed able to guarantee the fault tolerance of scientific workflow execution and improve cloud resource utilization.

A fault‐tolerant workflow management system with Quality‐of‐Service‐aware scheduling for scientific workflows in cloud computing

SummaryCloud computing provides solutions to many scientific and business applications. Large‐scale scientific applications, which are structured as scientific workflows, are evaluated through cloud computing. In this paper, we proposed a Quality‐of‐Service‐aware fault‐tolerant workflow management system (QFWMS) for scientific workflows in cloud computing. We have considered two real‐time scientific workflows, i.e., Montage and CyberShake, for an evaluation of the proposed QFWMS. The results of the proposed QFWMS scheduling were evaluated through simulation environment WorkflowSim and compared with three well‐known heuristic scheduling policies: (a) minimum completion time (MCT), (b) Maximum‐minimum (Max‐min), and (c) Minimum‐minimum (Min‐min). By considering Montage scientific workflow, the proposed QFWMS reduces the make‐span 8.86%, 8.94%, and 5.53% compared with existing three heuristic policies. Similarly, the proposed QFWMS reduces the cost 6.19%, 3.52%, and 3.60% compared with existing three heuristic policies. Likewise, by considering CyberShake scientific workflow, the proposed QFWMS reduces the make‐span 19.54%, 21.41%, and 25.71% compared with existing three heuristic policies. Similarly, the proposed QFWMS reduces the cost 8.78%, 8.40%, and 8.61% compared with existing three heuristic policies. More so, for QFWMS, SLA is neither violated for time constraints nor for cost constraints. While for MCT, Max‐min and Min‐min scheduling policies, SLA is violated 32, 37, and 23 times, respectively. Conclusively, the proposed QFWMS scheduling and management system is one of the significant workflow management systems for execution and management of scientific workflows in cloud computing.

Fuzzy Theory-Based Data Placement for Scientific Workflows in Hybrid Cloud Environments

In hybrid cloud environments, reasonable data placement strategies are critical to the efficient execution of scientific workflows. Due to various loads, bandwidth fluctuations, and network congestions between different data centers as well as the dynamics of hybrid cloud environments, the data transmission time is uncertain. Thus, it poses huge challenges to the efficient data placement for scientific workflows. However, most of the traditional solutions for data placement focus on deterministic cloud environments, which lead to the excessive data transmission time of scientific workflows. To address this problem, we propose an adaptive discrete particle swarm optimization algorithm based on the fuzzy theory and genetic algorithm operators (DPSO-FGA) to minimize the fuzzy data transmission time of scientific workflows. The DPSO-FGA can rationally place the scientific workflow data while meeting the requirements of data privacy and the capacity limitations of data centers. Simulation results show that the DPSO-FGA can effectively reduce the fuzzy data transmission time of scientific workflows in hybrid cloud environments.

Characterizing, Modeling, and Accurately Simulating Power and Energy Consumption of I/O-intensive Scientific Workflows

While distributed computing infrastructures can provide infrastructure-level techniques for managing energy consumption, application-level energy consumption models have also been developed to support energy-efficient scheduling and resource provisioning algorithms. In this work, we analyze the accuracy of a widely-used application-level model that has been developed and used in the context of scientific workflow executions. To this end, we profile two production scientific workflows on a distributed platform instrumented with power meters. We then conduct an analysis of power and energy consumption measurements. This analysis shows that power consumption is not linearly related to CPU utilization and that I/O operations significantly impact power, and thus energy, consumption. We then propose a power consumption model that accounts for I/O operations, including the impact of waiting for these operations to complete, and for concurrent task executions on multi-socket, multi-core compute nodes. We implement our proposed model as part of a simulator that allows us to draw direct comparisons between real-world and modeled power and energy consumption. We find that our model has high accuracy when compared to real-world executions. Furthermore, our model improves accuracy by about two orders of magnitude when compared to the traditional models used in the energy-efficient workflow scheduling literature.

GMTA: A Geo-Aware Multi-Agent Task Allocation Approach for Scientific Workflows in Container-Based Cloud

Scientific workflow scheduling is one of the most challenging problems in cloud computing because of the large-scale computing tasks and massive data volumes involved. A cloud system is a distributed system that follows the on-demand resource provisioning and pay-per-use billing model. Therefore, practical scheduling approaches are essential for good workflow performance and low overheads. This paper proposes a novel workflow allocation approach, the Geo-aware Multiagent Task Allocation Approach (GMTA), which aims to optimize large-scale scientific workflow execution in container-based clouds. GMTA is an agent-based workflow allocation method that includes a market-like agent negotiation mechanism and a dynamic workflow restructuring strategy. It decreases workflow makespans and traffic overheads by reasonable task replications. Furthermore, the performance of GMTA is verified on real scientific workflows in the CloudSim environment.

A Data and Task Co-Scheduling Algorithm for Scientific Cloud Workflows

Cloud computing has emerged as a promising computational infrastructure for cost-efficient workflow execution by provisioning on-demand resources in a pay-as-you-go manner. While scientific workflows require accessing community-wide resources, they usually need to be performed in collaborative cloud environments composed of multiple datacenters. Although such environments facilitate scientific collaboration, the movements of input and intermediate datasets across geographically distributed datacenters may cause intolerable latency that would hinder efficient execution of large-scale data-intensive scientific workflows. To address the problem, in this article we propose a novel multi-level K-cut graph partitioning algorithm to minimize the volume of data transfer across datacenters while satisfying load balancing and fixed data constraints. The algorithm first contracts the fixed input datasets in the same datacenter and their consuming tasks, and coarsens the contracted graph to a predefined scale in a level-by-level manner. Then, a K-cut algorithm is used to partition the resulted graph into K parts such that the cut size is minimized. After that, the partitioned graph is projected back to the original workflow graph, during which the load balancing constraint is maintained. We evaluate our algorithm using three real-world workflow applications and the results demonstrate that the proposed algorithm outperforms other state-of-the-art algorithms.

Adaptive Resource Allocation and Consolidation for Scientific Workflow Scheduling in Multi-Cloud Environments

The emerging multi-cloud environments (MCEs) empower the execution of large-scale scientific workflows (SWs) with sufficient resource provisioning. However, due to complex task dependencies in SWs and various cost-performance of cloud resources, the SW scheduling in MCEs faces huge challenges. To address these challenges, we propose an Online Workflow Scheduling algorithm based on Adaptive resource Allocation and Consolidation (OWS-A2C). In OWS-A2C, the deadline reassignment is first executed for SW tasks based on the execution performance of instance resources, which enhances resource utilization from a local perspective when executing an SW. Next, the execution instances are allocated and consolidated according to the performance requirements of multiple SWs, which improves resource utilization and reduces the total costs of executing multiple SWs from a global perspective. Finally, the SW tasks are dynamically scheduled to the execution instances with the earliest-deadline-first (EDF) discipline and completed before their sub-deadlines. The extensive simulation experiments are conducted to demonstrate the effectiveness of the proposed OWS-A2C on SW scheduling in MCEs, which outperforms three baseline scheduling methods with higher resource utilization and lower execution costs under deadline constraints.

Design of Energy Aware Scheduling Algorithm for Executing Scientific Workflows in Cloud

The usage of cloud computing and its resources for the execution of scientific workflow is a rapidly increasing demand. The Scientific applications are generally large in scale; even a single scientific workflow includes more number of complex tasks. Execution of these tasks can be made successful only by deploying it in the cloud virtual machines, because only cloud environment can only provide very large number of computing assets. In cloud, every processing resource is given as Virtual Machine. Any scientific workflow deployed in the cloud needs large number of virtual machines so; huge amount of computational energy is spent by the virtual machines to execute multifaceted scientific workflows. Hence there arises the need to utilize the cloud resources in an energy efficient way. Also, if the virtual machines are planned to schedule in an energy efficient manner there is an increase of makepsan of the workflow which is going to be an important parameter for completing the workflow within the deadline. So, the need for executing scientific workflows in energy efficient way with reduced makespan becomes a major issue among the researchers. It also becomes very challenging task to executing a scientific workflow in within the given deadline of a task in the given workflow. To address these issues, a new Energy Aware workflow scheduling algorithm is proposed and designed with improved makespan for the execution of different scientific applications in cloud environment.

Data-Oriented Scheduling with Dynamic-Clustering Fault-Tolerant Technique for Scientific Workflows in Clouds

Cloud computing is one of the most prominent parallel and distributed computing paradigm. It is used for providing solution to a huge number of scientific and business applications. Large scale scientific applications which are structured as scientific workflows are evaluated through cloud computing. Scientific workflows are data-intensive applications, as a single scientific workflow may consist of hundred thousands of tasks. Task failures, deadline constraints, budget constraints and improper management of tasks can also instigate inconvenience. Therefore, provision of fault-tolerant techniques with data-oriented scheduling is an important approach for execution of scientific workflows in Cloud computing. Accordingly, we have presented enhanced data-oriented scheduling with Dynamic-clustering fault-tolerant technique (EDS-DC) for execution of scientific workflows in cloud computing. We have presented data-oriented scheduling as a proposed scheduling technique. We have also equipped EDS-DC with Dynamic-clustering fault-tolerant technique. To know the effectiveness of EDS-DC, we compared its results with three well-known enhanced heuristic scheduling policies referred to as: (a) MCT-DC, (b) Max-min-DC, and (c) Min-min-DC. We considered scientific workflow of CyberShake as a case study, because it contains most of the characteristics of scientific workflows such as integration, disintegration, parallelism, and pipelining. The results show that EDS-DC reduced make-span of 10.9% as compared to MCT-DC, 13.7% as compared to Max-min-DC, and 6.4% as compared to Min-min-DC scheduling policies. Similarly, EDS-DC reduced the cost of 4% as compared to MCT-DC, 5.6% as compared to Max-min-DC, and 1.5% as compared to Min-min-DC scheduling policies. These results in respect of make-span and cost are highly significant for EDS-DC as compared with above referred three scheduling policies. The SLA is not violated for EDS-DC in respect of time and cost constraints, while it is violated number of times for MCT-DC, Max-min-DC, and Min-min-DC scheduling techniques.

A Novel Completion-Time-Minimization Scheduling Approach of Scientific Workflows Over Heterogeneous Cloud Computing Systems

The cloud computing paradigm provides an ideal platform for supporting large-scale scientific-workflow-based applications over the internet. However, the scheduling and execution of scientific workflows still face various challenges such as cost and response time management, which aim at handling acquisition delays of physical servers and minimizing the overall completion time of workflows. A careful investigation into existing methods shows that most existing approaches consider static performance of physical machines (PMs) and ignore the impact of resource acquisition delays in their scheduling models. In this article, the authors present a meta-heuristic-based method to scheduling scientific workflows aiming at reducing workflow completion time through appropriately managing acquisition and transmission delays required for inter-PM communications. The authors carry out extensive case studies as well based on real-world commercial cloud sand multiple workflow templates. Experimental results clearly show that the proposed method outperforms the state-of-art ones such as ICPCP, CEGA, and JIT-C in terms of workflow completion time.

ECOS: An efficient task-clustering based cost-effective aware scheduling algorithm for scientific workflows execution on heterogeneous cloud systems

Cloud Computing provides an attractive execution environment for scientific workflow execution. However, due to the increasingly high charge cost of using cloud service, cost minimization for workflows execution on cloud systems has become a crucial issue. Traditional work are adopting the sophisticated scheduling techniques to address such issue. Differently, this paper has proposed an efficient task-clustering based cost-effective aware scheduling algorithm (ECOS) to minimize the cost without comprising the deadline constraint. First, with respect to the characteristics of multi-type workflows, cloud heterogeneity and cost model, we have formulated the problem of task-clustering to simplify the structure of workflows and workflow scheduling to minimize cost within the deadline constraint. Then, we have devised ECOS with two key steps: (1) vertical clustering is with the time consideration that selectively merges the sequential tasks to reduce the transferring time within the workflow; (2) horizontal clustering and greedy allocation is to aggregate the parallel tasks and greedily allocate resources to that tasks with the aim of minimizing cost within deadline. Last, we have conducted the experiment that compare with well-known task-clustering based algorithms via WorkflowSim platform. The results have demonstrated that ECOS can efficiently merge tasks and minimize the total cost without comprising the deadline constraint both in small and large datasets. Moreover, we have discussed the ECOS in terms of various schedulers and number of tasks to validate the performance of ECOS.

Effective data placement for scientific workflows in mobile edge computing using genetic particle swarm optimization

SummaryMobile edge computing (MEC) necessitates cost‐effective deployment for executing scientific workflows with different tasks and datasets, which provides computing, storage and network control at the network edge. However, the execution of scientific workflows in MEC results in heavy costs of data placement including data transmission and data storage. Although there are solutions for data placement in traditional cloud computing, they cannot effectively respond to the latency‐sensitive property of scientific workflows, which leads to the excessive costs of data placement. To cope with this problem, we combine the advantages of MEC and cloud computing and propose a genetic algorithm particle swarm optimization (GAPSO) based method to explore the optimal strategy of data placement for scientific workflows in MEC. First, a unified model of data placement is designed to explore a cost‐effective strategy, which considers the different characteristics between MEC and cloud computing as well as the impact of latency constraint on transmission costs. Next, the advantages of genetic algorithm (GA) and particle swarm optimization (PSO) are integrated to optimize the proposed model, which utilities the fast convergence of PSO and the crossover and mutation operations of GA. Simulations using real‐world scientific workflows show the effectiveness of the proposed method for reducing data placement costs in MEC.