Workflows In Cloud Environment Research Articles

PPTS-PSO: a new hybrid scheduling algorithm for scientific workflow in cloud environment

PPTS-PSO: a new hybrid scheduling algorithm for scientific workflow in cloud environment

Hybrid collaborative multi-objective fruit fly optimization algorithm for scheduling workflow in cloud environment

Scheduling the complex workflows in cloud environment have drawn enormous attentions because the distinct features of the cloud resources. Most of the previous approaches ignored the multiple conflicting objectives of workflow scheduling and resources provisioning. In this paper, a novel hybrid collaborative multi-objective fruit fly optimization algorithm (HCMFOA) is developed to optimize both the execution time and cost. In the proposed HCMFOA, a reference points-based cluster strategy is introduced to dynamic divide the swarm into multiple sub-swarms. Moreover, a hybrid initial strategy is designed based on non-linear weight vector and two assignment rules of tasks to initialize the location of all the fruit flies in the problem space. In the collaborative smell-based foraging, three effective problem-specific neighborhood operators are employed to collaborative explore the global scope. In multi-objective vision-based foraging, the sub-swarms based crossover operator is designed to perform exploitation in local region. Finally, an extensive computational experiment is conducted to validate the performance of HCMFOA. The statistical results reveal that HCMFOA significantly outperforms the existing state-of-the-art approaches.

Energy‐aware autoscaling for scientific workflow in cloud environment

SummaryThe devices used for information and communication purpose consume an increasing amount of energy. As a result, the datacenters consume a lot of energy. Due to the expanding use of cloud services, there is considerable interest in reducing the energy consumption of datacenters. Effective resource utilization techniques and reduction in energy consumption are two crucial metrics in today's cloud computing environment. Many issues come along with contradicting objectives, viz., minimizing energy consumption raising the expenses of service supply. Autoscaling features of the cloud can bundle these issues. The major hurdle for effective resource scalability is the lack of proper management of energy supply to devices of servers in the datacenter. The main objective of the article is to supply energy to devices as per the allocated job‐size on the interaction of complex energy, to increase the energy utilization of servers. Also, using renewable energy (RE) to supplement the operational energy of the cloud datacenter. In this direction, the article presents a predator‐prey‐based mathematical model for energy scaling that leverages the accessibility of RE source to decrease the high energy demands of the datacenter. It is formalized using ecological concepts of metabolism and allometric scaling. An allometric energy scaling (AES) algorithm is presented. The performance and evaluation of the proposed algorithm are implemented with the Cloudsim 4.0 simulator. The results are compared with the HEFT algorithm, and AES algorithm gives slightly better results in the case of optimum energy consumption and processing cost compared with the HEFT algorithm.

Approaches for containerized scientific workflows in cloud environments with applications in life science

Containers are gaining popularity in life science research as they provide a solution for encompassing dependencies of provisioned tools, simplify software installations for end users and offer a form of isolation between processes. Scientific workflows are ideal for chaining containers into data analysis pipelines to aid in creating reproducible analyses. In this article, we review a number of approaches to using containers as implemented in the workflow tools Nextflow, Galaxy, Pachyderm, Argo, Kubeflow, Luigi and SciPipe, when deployed in cloud environments. A particular focus is placed on the workflow tool’s interaction with the Kubernetes container orchestration framework.

Design of a model based engineering deep learning scheduler in cloud computing environment using Industrial Internet of Things (IIOT)

The Industrial Internet of Things (IIoT) integrated with cloud computing resources offers effective advancements in the field of industrial automation with open connectivity and emergent computing. With such advancements, the supplementary services evolves in order to augment the automation process in manufacturing and production industries. However, the integration and transformation of cloud and IIoT environment poses serious limitation on reliability and computational performance. It is very necessary to develop a framework that handles and provides a bridge between the rapid input acquisitions from an IIoT device and rapid data processing or storage by the cloud. In this paper, model based engineering (MBE) model is used to control the workflow in cloud environment, which is integrated with a deep learning algorithm named recurrent neural network (RNN) algorithm. The RNN is utilized to control the future workload prediction and supply necessary resources to balance the workflow in cloud computing environment. The entire predictions are carried out based on the input data acquisition rate from the IIoT devices. The storage or processing of the input data is carried out by the Virtual Machines presents in cloud environment. The resources allocation and scheduling of workload is carried out in the control plane using RNN that optimizes the workflow. The RNN is resumed to solve iteratively the task of optimization. The experiments are conducted to test the efficacy of the model and the results are verified in terms of total energy consumption, task response time, and VM Utilization. The experiments are conducted with same/different types of IIoT devices. The simulation results reveal the improved efficacy of the MBE–RNN model than existing deep learning and MBE schedulers. The results show that the MBE–RNN acquires improved processing ability with reduced task response time and energy utilization.

Efficient Dynamic Scheduling Algorithm for Performance evaluation of Workflow in cloud Environment

Efficient Dynamic Scheduling Algorithm for Performance evaluation of Workflow in cloud Environment

Multi-objective scheduling strategy for scientific workflows in cloud environment: A Firefly-based approach

Cloud computing is a distributed computing paradigm, that provides infrastructure and services to the users using the pay-as-you-use billing model. With the increasing demands and diversity of the scientific workflows, the cloud providers face a fundamental issue of resource provisioning and load balancing. Although, the workflow scheduling in the cloud environment is extensively studied, however, most of the strategies ignore to consider the multiple conflicting objectives of the workflows for scheduling and resource provisioning. To address the above-mentioned issues, in the paper, we introduce a new workflow scheduling strategy using the Firefly algorithm (FA) by considering multiple conflicting objectives including workload of cloud servers, makespan, resource utilization, and reliability. The main purpose of the FA is to find a suitable cloud server for each workflow that can meet its requirements while balancing the loads and resource utilization of the cloud servers. In addition, a rule-based approach is designed to assign the tasks on the suitable VM instances for minimizing the makespan of the workflow while meeting the deadline. The proposed scheduling strategy is evaluated over Google cluster traces using various simulation runs. The control parameters of the FA are also thoroughly investigated for better performance. Through the experimental analysis, we prove that the proposed strategy performs better than the state-of-the-art-algorithms in terms of different Quality-of-Service (QoS) parameters including makespan, reliability, resource utilization and loads of the cloud servers.

An Algorithmic Analysis on Load Balancing and Workflow Scheduling in Cloud

Cloud computing is a paradigm for facilitating omnipresent, comfortable, on-demand network access to shared provisions of configurable computing resources like servers, applications, networks, and services that can be quickly provisioned and discharged with minimal management effort or service provider intercommunication. Data centers are the set of hosts that runs Virtual Machines. The broker submits the requests which are provisioned to the data centers that submit the task. The task or an application that runs is known as cloudlet. The term “Cloud computing” is to deliver the hosted services on the internet. Each “cloudlet” requires one or more computing resources from the cloud to complete its job. The executions of these sub-tasks use a scheduling algorithm in a particular fashion. Hence, one of the most important issues is that executing a large number of workflow in cloud environment and consumption of resources. The work proposes literature about the scheduling algorithm that resolves the conflict in the allocation of resources and manages the resource provisioning at dynamic to each task in any workflow. This paper proposes an analysis of proposed schemes with different workflow by executing the scheduling algorithm for the cloud workflow. This paper is presented with their objectives, parameters, and limitations and finally concluded with research directions.

Minimizing Monetary Costs for Deadline Constrained Workflows in Cloud Environments

As one of the latest market-oriented resource provisioning paradigms, cloud computing has been widely adopted by a growing number of consumers due to its powerful computing ability and storage ability. Although cloud computing can achieve effective cost reduction and convenience enhancement in the development of large-scale applications, it results in a complex cost optimization problem for data-dependent tasks represented by a workflow. All tasks in a workflow should be scheduled according to a proper strategy such that the cost is minimized and the precedence constraints and timing requirements are satisfied. In this paper, we study the cost optimization problem of deadline constrained workflows on cloud computing, and propose two list scheduling algorithms named Look-back Workflow Scheduling (LBWS) and Structure Aware Workflow Scheduling (SAWS) to solve the problem. LBWS distributes the deadline over the workflow as sub-deadlines to tasks in different levels, and schedules the tasks according to their priorities to the resources which meet their sub-deadlines and the best time-cost trade off requirements. Compared with LBWS, SAWS considers tasks allocated to the same level at a time and provisions resources with minimum cost to these tasks. Experiments on scientific workflow applications with different data and computational characteristics are conducted to show that, the proposed approaches can achieve better performance in terms of success rate and monetary cost.

A Survey on Scheduling Strategies for Workflows in Cloud Environment and Emerging Trends

Workflow scheduling is one of the challenging issues in emerging trends of the distributed environment that focuses on satisfying various quality of service (QoS) constraints. The cloud receives the applications as a form of a workflow, consisting of a set of interdependent tasks, to solve the large-scale scientific or enterprise problems. Workflow scheduling in the cloud environment has been studied extensively over the years, and this article provides a comprehensive review of the approaches. This article analyses the characteristics of various workflow scheduling techniques and classifies them based on their objectives and execution model. In addition, the recent technological developments and paradigms such as serverless computing and Fog computing are creating new requirements/opportunities for workflow scheduling in a distributed environment. The serverless infrastructures are mainly designed for processing background tasks such as Internet-of-Things (IoT), web applications, or event-driven applications. To address the ever-increasing demands of resources and to overcome the drawbacks of the cloud-centric IoT, the Fog computing paradigm has been developed. This article also discusses workflow scheduling in the context of these emerging trends of cloud computing.

Coupling Factor and Cost Based Task Clustering Method to Optimize Task Clustering For Scientific Workflows in Cloud Environment

Scientific workflows are large scale loosely coupled submissions that are used by Computational Scientists. They are composed of multiple tasks with dependencies between them and are composed of many fine granular tasks. Task clustering is an optimization method that combines multiple tasks into a single job such that task execution time and system overhead is reduced and thus the whole performance is improved in a cloud environment. Though existing task clustering algorithms has significantly reduced the System overhead, yet dependencies among the tasks are not well-thought-out. This work examines the features of task by which the tasks can be clustered and developed proficient task clustering algorithm. In this work two task clustering ideas were proposed namely Horizontal Coupling Factor (HCF) based clustering and Horizontal Processing Cost (HPC) based Task Clustering. Next, the proposed algorithm have been evaluated and tested for various real world applications and the experiment results shows that the proposed approach suits best for data intensive and Compute intensive applications. The obtained results showed that the HCF and HPC task clustering strategies can significantly improve the performance by reducing the task execution time and inter task Communication delay.

A Hybrid Algorithm for Scheduling Scientific Workflows in Cloud Computing

Cloud computing has become the main source for executing scientific experiments. It is an effective technique for distributing and processing tasks on virtual machines. Scientific workflows are complex and demand efficient utilization of cloud resources. Scheduling of scientific workflows is considered as NP-complete. The problem is constrained by some parameters such as Quality of Service (QoS), dependencies between tasks and users’ deadlines, etc. There exists a strong literature on scheduling scientific workflows in cloud environments. Solutions include standard schedulers, evolutionary optimization techniques, etc. This article presents a hybrid algorithm for scheduling scientific workflows in cloud environments. In the first phase, the algorithm prepares tasks lists for PSO algorithm. Bottleneck tasks are processed on high priority to reduce execution time. In the next phase, tasks are scheduled with the PSO algorithm to reduce both execution time and monetary cost. The algorithm also monitors the load balance to efficiently utilize cloud resources. Benchmark scientific workflows are used to evaluate the proposed algorithm. The proposed algorithm is compared with standard PSO and specialized schedulers to validate the performance. The results show improvement in execution time, monetary cost without affecting the load balance as compared to other techniques.

An intelligent water drops-based approach for workflow scheduling with balanced resource utilisation in cloud computing

The problem of finding optimal solutions for scheduling scientific workflows in cloud environment has been thoroughly investigated using various nature-inspired algorithms. These solutions minimise the execution time of workflows, however may result in severe load imbalance among Virtual Machines (VMs) in cloud data centres. Cloud vendors desire the proper utilisation of all the VMs in the data centres to have efficient performance of overall system. Thus, load balancing of VMs becomes an important aspect while scheduling tasks in cloud environment. In this paper, we propose an approach based on Intelligent Water Drops (IWD) algorithm to minimise the execution time of workflows while balancing the resource utilisation of VMs in cloud computing environment. The proposed approach is compared with a variety of well-known heuristic and meta-heuristic techniques using three real-time scientific workflows, and experimental results show that the proposed algorithm performs better than these existing techniques in terms of makespan and load balancing.

Multi Criteria based Resource Score Heuristic for Cloud Workflow Scheduling

Abstract Scheduling scientific workflows modelled by Directed Acyclic Graphs (DAG) is an NP complete problem. Cloud computing provides reliable Quality of Service defined in terms of Service Level Agreements (SLA). To schedule scientific workflows in cloud environment, where resources are shared, it is important to manage the cloud resources efficiently by maximizing utilization. The dynamic nature of cloud resources, due to sharing, heterogeneity, virtualization and workload variations offer a host of challenges in terms of resource availability and performance. This may have a significant impact on task execution times and data transfer times, thus introducing delay in overall execution time called makespan, In this paper, Multi Criteria based Resource Score Heuristic for Cloud Workflow Scheduling is proposed, with an objective of minimizing makespan considering the probability of temporal availability of resources in a cloud computing environment. To choose the best virtual machine with optimal value of maximum resource availability and minimum task execution time, a method of compensatory aggregation of conflicting criteria, is used for scoring each resource. The simulation results demonstrate that the proposed heuristic, can generate schedules with better makespan minimization and is found to be more reliable since resource availability factor is considered while mapping tasks to virtual machines.

MOPT: list-based heuristic for scheduling workflows in cloud environment

Cloud computing is a popular and widely adopted computing platform for the execution of scientific workflows as it provides flexible infrastructure and offers access to collection of autonomous heterogeneous resources. Effective scheduling of computationally complex workflows which contain many interconnected tasks is a complex problem and becomes more challenging in cloud environment. Optimal solutions can be obtained by considering not only the heterogeneity of computation costs involved, but also by taking into account the communication costs among the tasks in a way that schedule length of the application is reduced. In this paper, we propose a list scheduling heuristic, namely minimal optimistic processing time (MOPT), with optimized duplication approach. The additional feature is introduced for the entry task and is applied only in scenarios in which duplication is more practical and effective. The prioritization phase of the proposed work is based on an optimistic processing time matrix that is used for ranking of the tasks. The algorithm has same time complexity as state-of-the-art existing algorithms, but notable improvements are acquired in terms of makespan and other performance evaluation parameters. Extensive experimental analysis of the proposed algorithm is carried out using synthesized graphs and graphs from the real-world applications. The results prove that MOPT achieves quality schedules with reduced makespans. As communication cost among the tasks grows higher, performance of the proposed algorithm becomes more effective, thus providing the evidence that the MOPT algorithm is well-suited for communication-intensive applications.

Energy Efficient Scheduling for Multiple Workflows in Cloud Environment

Cloud computing makes utility computing possible with pay as you go model. It virtualizes the systems by polling and sharing the resources, thus we need to handle more than one workflow at the same time. Workflow is the standard to represent compute intensive applications in scientific and engineering domain. Hence, in this article, the authors presented the scheduling heuristic for multiple workflows running parallel in the cloud environment with the aim to reduce the energy consumption as it is one of the major concerns of cloud data centers along with the execution performance. In the proposed approach, first clustering is performed to minimize the energy consumption and execution time during communication corresponding to precedence constraint tasks. Then cluster are scheduled is on the best available energy efficient resources. Finally, DVFS is applied in order to reduce energy consumption further when the nodes are in the idle and communication stage. The simulation has been performed on CloudSim and the results show the reduction in energy consumption by up to 42%.

Enhanced Resource Provisioning Strategies for Scientific Workflows in Cloud Environment: A Survey

Enhanced Resource Provisioning Strategies for Scientific Workflows in Cloud Environment: A Survey

DSM: a dynamic scheduling method for concurrent workflows in cloud environment

Cloud computing, emerged as a commercial service model, has been widely concerned in both industry and academia. Massive workflow applications could be performed simultaneously on the cloud platforms, which significantly benefits from the elasticity and convenience of cloud computing. However, it is still a challenge to schedule virtualized resources for the concurrent workflows in cloud environment, with limited high-performance resources in a timesaving and efficient manner. In view of this challenge, a dynamic scheduling method for concurrent workflows, named as DSM, in cloud environment is proposed to satisfy the various resource requirements of the workflows. Technically, a time overhead model for the workflows and a resource utilization model for cloud datacenter are presented. Then a relevant dynamic scheduling method is designed based on critical path lookup, which aims at minimizing the makespan of workflows, and maximizing the resource utilization of the datacenter during the execution of the workflows. Extensive experimental evaluations demonstrate the efficiency and effectiveness of our proposed method.

Using MinMax-Memory Claims to Improve In-Memory Workflow Computations in the Cloud

In this paper, we consider to improve scientific workflows in cloud environments where data transfers between tasks are performed via provisioned in-memory caching as a service, instead of relying entirely on slower disk-based file systems. However, this improvement is not free since services in the cloud are usually charged in a “pay-as-you-go” model. As a consequence, the workflow tenants have to estimate the amount of memory that they would like to pay. Given the intrinsic complexity of the workflows, it would be very hard to make an accurate prediction, which would lead to either oversubscription or undersubscription, resulting in unproductive spending or performance degradation. To address this problem, we propose a concept of minmax memory claim (MMC) to achieve cost-effective workflow computations in in-memory cloud computing environments. The minmax-memory claim is defined as the minimum amount of memory required to finish the workflow without compromising its maximum concurrency. With the concept of MMC, the workflow tenants can achieve the best performance via in-memory computing while minimizing the cost. In this paper, we present the procedure of how to find the MMCs for those workflows with arbitrary graphs in general and develop optimal efficient algorithms for some well-structured workflows in particular. To further show the values of this concept, we also implement these algorithms and apply them, through a simulation study, to improve deadlock resolutions in workflow-based workloads when memory resources are constrained.

Data-Locality Aware Scientific Workflow Scheduling Methods in HPC Cloud Environments

Efficient data-aware methods in job scheduling, distributed storage management and data management platforms are necessary for successful execution of data-intensive applications. However, research about methods for data-intensive scientific applications are insufficient in large-scale distributed cloud and cluster computing environments and data-aware methods are becoming more complex. In this paper, we propose a Data-Locality Aware Workflow Scheduling (D-LAWS) technique and a locality-aware resource management method for data-intensive scientific workflows in HPC cloud environments. D-LAWS applies data-locality and data transfer time based on network bandwidth to scientific workflow task scheduling and balances resource utilization and parallelism of tasks at the node-level. Our method consolidates VMs and consider task parallelism by data flow during the planning of task executions of a data-intensive scientific workflow. We additionally consider more complex workflow models and data locality pertaining to the placement and transfer of data prior to task executions. We implement and validate the methods based on fairness in cloud environments. Experimental results show that, the proposed methods can improve performance and data-locality of data-intensive workflows in cloud environments.