Scientific Workflows In Cloud Research Articles

A multi-objective crow search algorithm for optimizing makespan and costs in scientific cloud workflows (CSAMOMC)

A multi-objective crow search algorithm for optimizing makespan and costs in scientific cloud workflows (CSAMOMC)

Optimal Data Placement for Scientific Workflows in Cloud

ABSTRACT The deployment of datasets in the heterogeneous cloud computing has received increasing attention in current research. However, due to their large sizes and the existence of private scientific datasets, finding an optimal data placement strategy remains a persistent problem. The primary goal of this work is to discover an optimum placement while satisfying the security demand (SD) at the lowest cost. To effectively address this problem, a security-based optimal workflow scheduling (OWS) is proposed for privacy-aware applications over data. During negotiation, the user can submit the SD to the cloud. This work is initialized with list-based heuristics with Particle Swarm Hybridized Red Deer (PSRD). The proposed system can assign tasks for the scientific workflow in the cloud according to the virtual machine (VM). The results show that the workflow schedule provides better security yielding good makespan than the conventional methods with minimum iteration suited for a cloud environment.

Cost effective hybrid genetic algorithm for scheduling scientific workflows in cloud under deadline constraint

Cloud has emerged as a convenient platform for executing complicated scientific applications from multiple disciplines by providing on-demand and scalable infrastructure on rental basis. Research and scientific community often opt for workflows to model these scientific applications Workflow scheduling has been extensively studied for decades with regard to grid and cluster computing, but few initiatives have been tailored for cloud. What's more, the previous work fails to incorporate the basic principles of IaaS clouds like pay-as-you-go model, elasticity, heterogeneity, dynamic provisioning and issues of VM's performance variation and acquisition delay besides other QoS requirements. This paper proposes a resource provisioning and scheduling strategy using genetic algorithm with the aim to optimise the overall execution cost while staying below the given deadline. The performance is further enhanced by using a high quality seed generated by predict earliest finish time (PEFT) algorithm which acts as a catalyst and helps the algorithm to converge faster. The proposed approach is simulated in WorkflowSim and evaluated using various well-known different sized realistic scientific workflows. The results validate the better performance of our approach over numerous state-of-art-algorithms.

A Provenance-based Execution Strategy for Variant GPU-accelerated Scientific Workflows in Clouds

A Provenance-based Execution Strategy for Variant GPU-accelerated Scientific Workflows in Clouds

A PSO task scheduling and IT2FCM fuzzy data placement strategy for scientific cloud workflows

Scientific workflows are appropriate for treating large volumes of data and leading large-scale experiments. They are time-consuming and resource-intensive applications that profit from operating within distributed platforms, like cloud computing. However, the cloud presents several challenges that must be addressed to employ workflow applications efficiently. Although the workflow scheduling problem has been broadly examined, it still attracts attention. Furthermore, in the literature, little research has been conducted taking into consideration the data placement process in parallel with the execution of task scheduling. Hence, this paper proposes a task scheduling and data placement strategy to reduce the number of data movements between cloud data centers. The suggested strategy uses an algorithm based on the fuzzy clustering method Interval Type-2 Fuzzy C-Means (IT2FCM) and the meta-heuristic optimization technique Particle Swarm Optimization (PSO) to minimize data movements throughout the workflow’s execution. The proposed strategy is evaluated through a simulation process using various well-known scientific workflows of different sizes. Additionally, the outcomes show that our strategy is more reliable than the most recent state-of-the-art techniques.

A hybrid algorithm for scheduling scientific workflows in IaaS cloud with deadline constraint

A hybrid algorithm for scheduling scientific workflows in IaaS cloud with deadline constraint

A heuristic-based task scheduling algorithm for scientific workflows in heterogeneous cloud computing platforms

Task scheduling problem in a cloud computing environment is one of the most important issues in this field. Because task schedulers should be aware of underlying platform heterogeneity, task interdependencies, and virtual machine (VM) variable configurations. An efficient task scheduling algorithm can potentially increase the efficiency of cloud computing, by choosing the appropriate virtual machine to do a specific task-scheduling problem, for dependent tasks on heterogeneous resources which is a well-known NP-Hard problem. To reduce minimum total execution time, makespan, in cloud computing many heuristics were presented in literature but lots of them suffer from low efficiency. In this article, a new task priority strategy and applying task duplication methods are proposed, for solving the task scheduling problem of the dependent tasks in heterogeneous cloud computing systems. The novelty of the current paper is to present a new list scheduling algorithm with a new task priority strategy and applying pertinent task duplication techniques. This paper uses optimistic cost table downward (OCTd) and optimistic cost table upward (OCTu) procedures to prioritize tasks in an efficient ordered list; then, it utilizes Heterogeneous Earliest Finish Time (HEFT)-duplication method for performing task duplication technique which significantly reduces makespan. To validate the proposal, we experimentally analyzed the proposed scheduling algorithm with different scientific workflows such as Molecular, LU-Like, FFT, and Montage datasets. The performance comparison of the novel heuristic scheduling algorithm against other existing approaches proved the superiority of proposed algorithm in terms of makespan, speedup, SLR, and efficiency which are prominent scheduling evaluation metrics.

Big data analytics in Industry 4.0 ecosystems

Big data analytics in Industry 4.0 ecosystems

Real-time and dynamic fault-tolerant scheduling for scientific workflows in clouds

Cloud computing has become a popular technology for executing scientific workflows. However, with a large number of hosts and virtual machines (VMs) being deployed, the cloud resource failures, such as the permanent failure of hosts (HPF), the transient failure of hosts (HTF), and the transient failure of VMs (VMTF), bring the service reliability problem. Therefore, fault tolerance for time-consuming scientific workflows is highly essential in the cloud. However, existing fault-tolerant (FT) approaches consider only one or two above failure types and easily neglect the others, especially for the HTF. This paper proposes a Real-time and dynamic Fault-tolerant Scheduling (ReadyFS) algorithm for scientific workflow execution in a cloud, which guarantees deadline constraints and improves resource utilization even in the presence of any resource failure. Specifically, we first introduce two FT mechanisms, i.e., the replication with delay execution (RDE) and the checkpointing with delay execution (CDE), to cope with HPF and VMTF, simultaneously. Additionally, the rescheduling (ReSC) is devised to tackle the HTF that affects the resource availability of the entire cloud datacenter. Then, the resource adjustment (RA) strategy, including the resource scaling-up (RS-Up) and the resource scaling-down (RS-Down), is used to adjust resource demands and improve resource utilization dynamically. Finally, the ReadyFS algorithm is presented to schedule real-time scientific workflows by combining all the above FT mechanisms with RA strategy. We conduct the performance evaluation with real-world scientific workflows and compare ReadyFS with five vertical comparison algorithms and three horizontal comparison algorithms. Simulation results confirm that ReadyFS is indeed able to guarantee the fault tolerance of scientific workflow execution and improve cloud resource utilization.

A fault‐tolerant workflow management system with Quality‐of‐Service‐aware scheduling for scientific workflows in cloud computing

SummaryCloud computing provides solutions to many scientific and business applications. Large‐scale scientific applications, which are structured as scientific workflows, are evaluated through cloud computing. In this paper, we proposed a Quality‐of‐Service‐aware fault‐tolerant workflow management system (QFWMS) for scientific workflows in cloud computing. We have considered two real‐time scientific workflows, i.e., Montage and CyberShake, for an evaluation of the proposed QFWMS. The results of the proposed QFWMS scheduling were evaluated through simulation environment WorkflowSim and compared with three well‐known heuristic scheduling policies: (a) minimum completion time (MCT), (b) Maximum‐minimum (Max‐min), and (c) Minimum‐minimum (Min‐min). By considering Montage scientific workflow, the proposed QFWMS reduces the make‐span 8.86%, 8.94%, and 5.53% compared with existing three heuristic policies. Similarly, the proposed QFWMS reduces the cost 6.19%, 3.52%, and 3.60% compared with existing three heuristic policies. Likewise, by considering CyberShake scientific workflow, the proposed QFWMS reduces the make‐span 19.54%, 21.41%, and 25.71% compared with existing three heuristic policies. Similarly, the proposed QFWMS reduces the cost 8.78%, 8.40%, and 8.61% compared with existing three heuristic policies. More so, for QFWMS, SLA is neither violated for time constraints nor for cost constraints. While for MCT, Max‐min and Min‐min scheduling policies, SLA is violated 32, 37, and 23 times, respectively. Conclusively, the proposed QFWMS scheduling and management system is one of the significant workflow management systems for execution and management of scientific workflows in cloud computing.

A parallel multi-objective genetic algorithm for scheduling scientific workflows in cloud computing

Recent developments in cloud computing have made it a powerful solution for executing large-scale scientific problems. The complexity of scientific workflows demands efficient utilization of cloud resources to satisfy user requirements. Scheduling of scientific workflows in a cloud environment is a challenge for researchers. The problem is considered as NP-hard. Some constraints such as a heterogeneous environment, dependencies between tasks, quality of service and user deadlines make it difficult for the scheduler to fully utilize available resources. The problem has been extensively studied in the literature. Different researchers have targeted different parameters. This article presents a multi-objective scheduling algorithm for scheduling scientific workflows in cloud computing. The solution is based on genetic algorithm that targets makespan, monetary cost, and load balance. The proposed algorithm first finds the best solution for each parameter. Based on these solutions, the algorithm finds the superbest solution for all parameters. The proposed algorithm is evaluated with benchmark datasets and comparative results with the standard genetic algorithm, particle swarm optimization, and specialized scheduler are presented. The results show that the proposed algorithm achieves an improvement in makespan and reduces the cost with a well load balanced system.

A Data and Task Co-Scheduling Algorithm for Scientific Cloud Workflows

Cloud computing has emerged as a promising computational infrastructure for cost-efficient workflow execution by provisioning on-demand resources in a pay-as-you-go manner. While scientific workflows require accessing community-wide resources, they usually need to be performed in collaborative cloud environments composed of multiple datacenters. Although such environments facilitate scientific collaboration, the movements of input and intermediate datasets across geographically distributed datacenters may cause intolerable latency that would hinder efficient execution of large-scale data-intensive scientific workflows. To address the problem, in this article we propose a novel multi-level K-cut graph partitioning algorithm to minimize the volume of data transfer across datacenters while satisfying load balancing and fixed data constraints. The algorithm first contracts the fixed input datasets in the same datacenter and their consuming tasks, and coarsens the contracted graph to a predefined scale in a level-by-level manner. Then, a K-cut algorithm is used to partition the resulted graph into K parts such that the cut size is minimized. After that, the partitioned graph is projected back to the original workflow graph, during which the load balancing constraint is maintained. We evaluate our algorithm using three real-world workflow applications and the results demonstrate that the proposed algorithm outperforms other state-of-the-art algorithms.

Protecting scientific workflows in clouds with an intrusion tolerant system

With the development of cloud computing technology, more and more scientific workflows are delivered to cloud platforms to complete. However, there are many threats in clouds due to the multi-tenant coexistence. In order to protect scientific workflows in clouds, the authors propose an intrusion tolerant scientific workflow system. In this system, the task executors containing multiple virtual machines are used for workflow sub-task execution to enhance reliability. Then lagged decision mechanism is presented to ensure uninterrupted workflow execution while checking the intermediate data, and assessing the confidence of these data. Inspired by moving target defence, they propose a dynamic task scheduling strategy based on resource circulation to periodically generate and recycle task executors, keeping the clean state of the workflow execution environment. Furthermore, temporary workflow intermediate data backup mechanism is presented, the stored intermediate data can be used for the re-execution of workflow sub-tasks with low confidence. Experiments are conducted in both the actual test environment based on OpenStack and the simulated test environment based on WorkflowSim toolkit. Experimental results demonstrate that the proposed system can effectively enhance intrusion tolerance of scientific workflows.

Scheduling Workflows With Composite Tasks: A Nested Particle Swarm Optimization Approach

Scientific cloud workflows enable the access to distributed computing resources in cloud environments for executing scientific computing applications. In the literature, most workflow scheduling models assume that each workflow task is mapped to only one service instance. But in computation and data-intensive applications, it is common that the computation resources provided by a single service instance are insufficient for some complicated tasks which contain several closely correlated sub-tasks. To manage such complicated workflows, this article devises a novel workflow model with composite tasks (cWFS). The model views a complicated task as a composite task and allows mapping multiple service instances to a composite task. The data transmission among sub-tasks of a composite task can also be addressed by the proposed model. To solve cWFS problem, we devise a nested particle swarm optimization (N-PSO) that utilizes two kinds of populations, i.e., the outer population and inner population. Since N-PSO is a bit time-consuming, we further devise a Fast version of N-PSO (FN-PSO), which can save more than 60 percent of running time compared with N-PSO. The proposed approaches are evaluated on five real-world workflow types. The experimental results verify that the proposed approaches can solve the new workflow model effectively.

Deep‐Q learning‐based heterogeneous earliest finish time scheduling algorithm for scientific workflows in cloud

SummaryThe complex and large‐scale scientific workflow applications are effectively executes on the cloud. The performance of cloud computing highly depends on the task scheduling. Optimal workflow scheduling is still a challenge that needs to be addressed due to the conflicting objectives and increasing demand for quality of service. Task scheduling is an NP‐hard problem due to its complexity. The newly introduced methods for resolving the problem of task scheduling are facing challenges to take the benefits of all aspects of cloud computing. In this article, we study the joint optimization of cost and makespan of scheduling workflows in infrastructure as a service clouds and propose a new workflow scheduling scheme using deep learning. In this scheme, a deep‐Q learning‐based heterogeneous earliest‐finish‐time (DQ‐HEFT) algorithm is developed, which closely integrates the deep learning mechanism with the task scheduling heuristic HEFT. The workflowsim simulator is used for the experiment of the real‐world and synthetic workflows. The experiment results demonstrate the efficiency of our proposed approach compared with existing algorithms. This technique can achieve significantly better makespan and speed metrics with a remarkably higher volume of data and can run faster compared with the existing workflow scheduling algorithms in cloud computing environment.

Design of Energy Aware Scheduling Algorithm for Executing Scientific Workflows in Cloud

The usage of cloud computing and its resources for the execution of scientific workflow is a rapidly increasing demand. The Scientific applications are generally large in scale; even a single scientific workflow includes more number of complex tasks. Execution of these tasks can be made successful only by deploying it in the cloud virtual machines, because only cloud environment can only provide very large number of computing assets. In cloud, every processing resource is given as Virtual Machine. Any scientific workflow deployed in the cloud needs large number of virtual machines so; huge amount of computational energy is spent by the virtual machines to execute multifaceted scientific workflows. Hence there arises the need to utilize the cloud resources in an energy efficient way. Also, if the virtual machines are planned to schedule in an energy efficient manner there is an increase of makepsan of the workflow which is going to be an important parameter for completing the workflow within the deadline. So, the need for executing scientific workflows in energy efficient way with reduced makespan becomes a major issue among the researchers. It also becomes very challenging task to executing a scientific workflow in within the given deadline of a task in the given workflow. To address these issues, a new Energy Aware workflow scheduling algorithm is proposed and designed with improved makespan for the execution of different scientific applications in cloud environment.

Data-Oriented Scheduling with Dynamic-Clustering Fault-Tolerant Technique for Scientific Workflows in Clouds

Cloud computing is one of the most prominent parallel and distributed computing paradigm. It is used for providing solution to a huge number of scientific and business applications. Large scale scientific applications which are structured as scientific workflows are evaluated through cloud computing. Scientific workflows are data-intensive applications, as a single scientific workflow may consist of hundred thousands of tasks. Task failures, deadline constraints, budget constraints and improper management of tasks can also instigate inconvenience. Therefore, provision of fault-tolerant techniques with data-oriented scheduling is an important approach for execution of scientific workflows in Cloud computing. Accordingly, we have presented enhanced data-oriented scheduling with Dynamic-clustering fault-tolerant technique (EDS-DC) for execution of scientific workflows in cloud computing. We have presented data-oriented scheduling as a proposed scheduling technique. We have also equipped EDS-DC with Dynamic-clustering fault-tolerant technique. To know the effectiveness of EDS-DC, we compared its results with three well-known enhanced heuristic scheduling policies referred to as: (a) MCT-DC, (b) Max-min-DC, and (c) Min-min-DC. We considered scientific workflow of CyberShake as a case study, because it contains most of the characteristics of scientific workflows such as integration, disintegration, parallelism, and pipelining. The results show that EDS-DC reduced make-span of 10.9% as compared to MCT-DC, 13.7% as compared to Max-min-DC, and 6.4% as compared to Min-min-DC scheduling policies. Similarly, EDS-DC reduced the cost of 4% as compared to MCT-DC, 5.6% as compared to Max-min-DC, and 1.5% as compared to Min-min-DC scheduling policies. These results in respect of make-span and cost are highly significant for EDS-DC as compared with above referred three scheduling policies. The SLA is not violated for EDS-DC in respect of time and cost constraints, while it is violated number of times for MCT-DC, Max-min-DC, and Min-min-DC scheduling techniques.

A Novel Workflow-Level Data Placement Strategy for Data-Sharing Scientific Cloud Workflows

Cloud computing can provide a more cost-effective way to deploy scientific workflows than traditional distributed computing environments such as cluster and grid. Due to the large size of scientific datasets, data placement plays an important role in scientific cloud workflow systems for improving system performance and reducing data transfer cost. Traditional task-level data placement strategy only considers shared datasets within individual workflows to reduce data transfer cost. However, it is obvious that task-level strategy is not necessarily good enough for the situation of multiple workflows at the workflow level. In this paper, a novel workflow-level data placement model is constructed, which regards multiple workflows as a whole. Then, a two-stage data placement strategy is proposed which first pre-allocates initial datasets to proper datacenters during workflow build-time stage, and then dynamically distributes newly generated datasets to appropriate datacenters during runtime stage. Both stages use an efficient discrete particle swarm optimization algorithm to place flexible-location datasets. Comprehensive experiments demonstrate that our workflow-level data placement strategy can be more cost-effective than its task-level counterpart for data-sharing scientific cloud workflows.

Securing the Intermediate Data of Scientific Workflows in Clouds With ACISO

Many scientific workflow applications are moving to clouds. A scientific workflow is a complicated scientific computing task consisting of many sub-tasks, and each sub-task execution can generate the intermediate data used for the successor sub-task execution. The correct execution of scientific workflows depends on the security of the intermediate data, which is transmitted frequently between virtual machines during the process of the workflow execution. In multi-tenant clouds, the intermediate data contains three attributes: availability, confidentiality and integrity. If the intermediate data is lost, stolen, or tampered with by malicious tenants, the intermediate data’s attribute will be damaged, causing workflow interruption, the leakage of secret information or incorrect workflow results. For these problems, we propose ACISO scheme to secure the intermediate data by improving its availability, confidentiality, and integrity. In the scheme, availability, confidentiality and integrity strategy pools are constructed by various erasure codes, encryption algorithms and hash functions, respectively. Then we present a security strategy optimal allocation model named SSOA, which aims to maximize the overall intermediate data security strength while meeting the constraints of the workflow makespan and storage overhead. Normally, a scientific workflow contains a large number of the intermediate data, so solving this model is NP hard. Therefore, we propose a heuristic solution to solve SSOA. The simulation results show that ACISO can effectively improve the availability, confidentiality, and integrity of the intermediate data of the scientific workflows.

Ориентированное на данные планирование с применением отказоустойчивого метода динамической кластеризации для поддержки потоков научных работ в облаках

Ориентированное на данные планирование с применением отказоустойчивого метода динамической кластеризации для поддержки потоков научных работ в облаках