Large-scale Scientific Workflows Research Articles

Notice of Violation of IEEE Publication Principles: Scientific Workflow Partitioning and Data Flow Optimization in Hybrid Clouds

Notice of Violation of IEEE Publication Principles “Scientific Workflow Partitioning and Data Flow Optimization in Hybrid Clouds” by Rubing Duan, Rick Siow Mong Goh, Zheng Qin and Young Liu in Pre-Prints for IEEE Transactions on Cloud Computing After careful and considered review of the content and authorship of this paper by a duly constituted expert committee, this paper has been found to be in violation of IEEE's Publication Principles. Due to the nature of this violation, reasonable effort should be made to remove all past references to this paper and it should not be used for research or citation The scheduling and execution of workflow applications are complex issues because of the dynamic and heterogeneous nature of distributed computing environments like hybrid clouds. While hybrid cloud computing environments provide good potential for achieving high performance and low economic cost, it also introduces a broad set of unpredictable overheads. This paper describes a novel approach to refine workflow structure and optimize intermediate data transfers without changing the scale of scheduled cloud service for large-scale scientific workflows containing thousands (or even millions) of tasks. These methods include pre- and post-partitioning of workflows, data flow optimization, static and dynamic optimization, as well as virtual single execution environment incorporated into a workflow management system.We demonstrate the effectiveness of our methods on the three real applications, Montage, Broadband, and WIEN2k, executed in a real hybrid cloud.

Efficient, economical and energy-saving multi-workflow scheduling in hybrid cloud

Benefiting from the flexible, scalable and secure environment, hybrid cloud can overcome the shortage of limited resources in private cloud to simultaneously execute large-scale scientific workflows. In hybrid cloud, privacy-sensitive tasks are not allowed to be executed on public resources, while non-sensitive tasks are unrestricted. As an NP-Complete problem, it is extraordinarily challenging to schedule multiple workflows efficiently, economically and energy-savingly under quality-of-service constraints. This paper models the hybrid-cloud-based privacy-aware multi-workflow scheduling as a tri-objective optimization problem that optimizes workflow-oriented total tardiness, private-cloud-oriented total energy consumption, and public-cloud-oriented total monetary cost. To the best of authors’ knowledge, few studies have been conducted on the tri-objective privacy-aware multi-workflow scheduling in hybrid cloud (PMWS-HC). To solve this problem, we dissect various factors involved during task scheduling and devise a novel Heuristic Scheduling Algorithm based on 9 Factors (HSA9Fs), which dynamically selects the workflows and tasks to be scheduled, and the corresponding VMs to execute them. To optimize the three conflicting objectives simultaneously, we propose a nested algorithm called MSIA, which first employs a Multi-objective Salp swarm algorithm to explore for the Pareto solutions, and then uses an Iterative greedy Algorithm to perform a refined search on individuals to obtain high-quality solutions. Extensive Medium-Small-Scale and Large-Scale simulation experiments show that both HSA9Fs and MSIA outperform state-of-the-art scheduling algorithms in several multi-objective performance metrics.

Multi-objective optimization for scientific workflow scheduling based on Performance-to-Power Ratio in fog–cloud environments

The ever-expanding processing of large-scale scientific workflow applications has increased power consumption and reliability costs. However, considering the complexity of computational data processing in fog–cloud computing, existing workflow scheduling techniques conceived for traditional distributed systems is limited and powerless. Therefore, for profit-driven cloud service providers, minimizing power consumption and maximizing scheduling reliability of scientific workflows is a pressing concern. In this paper, we model the scientific workflow scheduling as a multi-objective optimization problem to compromise the conflicting energy consumption and scheduling reliability solution in two steps. The first step places the tasks with lower computational requirements on fog resources and computationally complex tasks on cloud resources, avoiding resources with high failure rates. In the second step, we adapt the Reliability-Aware stepwise Performance-to-Power Ratio procedure to further reduce energy consumption by sampling the machine’s utilization levels with a distinguished Performance-to-Power Ratio (PPR). The PPR is defined as the calculation of the number of transactions performed at a specific time divided by the active power consumed. Simulation results under synthesized and real-world workflow applications demonstrated that dis-RMEE achieved considerably higher reliability values under the minimized energy constraint compared with some opponent workflow mapping algorithms.

Scoring and Dynamic Hierarchy-Based NSGA-II for Multiobjective Workflow Scheduling in the Cloud

Cloud computing becomes a promising technology to reduce computation cost by providing users with elastic resources and application-deploying environments as a pay-per-use model. More scientific workflow applications have been moved or are being migrated to the cloud. Scheduling workflows turns to the main bottleneck for increasing resource utilization and quality of service (QoS) for users. This work formulates workflow scheduling as multiobjective optimization problems and proposes a Scoring and Dynamic Hierarchy-based NSGA-II (Nondominated Sorting Genetic Algorithm II), called SDHN for short, to minimize both makespan and cost of workflow execution. First, a scoring criterion is developed to calculate the total score for each individual during population updating, which is used as a quantitative index to evaluate the dominance degree of individuals among the whole population. Hence, SDHN can distinguish individuals within the same dominance level and target its search toward the directions of elite solutions as their different dominance degrees and accordingly improve search efficiency. Second, a population-based dynamic hierarchical structure (HS) and its evolutionary rules are presented to update HS by comparing each child with all parental individuals from bottom to up until finding a proper dominant level. Since traversing all HS levels is not needed in most cases, the number of individual comparisons is reduced and SDHN&#x2019;s updating efficiency is greatly improved, especially for large-scale and complex applications. Third, to guarantee its converging to the near-optimal solutions, adaptive adjustment strategies (AASs) are designed to prevent the search from falling into local optima or diverging by checking the number of individuals at the highest HS level and then modifying the relevant genetic operations to guide the evolutionary process to approach the global Pareto Front. Extensive experiments are conducted to verify SDHN, and the results show that it outperforms the existing algorithms in the quality and diversity of resulting solutions as well as convergence time. <i>Note to Practitioners</i>&#x2014;Most scientific applications are computation and/or data-intensive and need large-scale or high-performance resources for their execution. More and more scientists use workflows to manage their applications, but how to efficiently run them in the cloud is a big challenge due to their large scale as well as the dynamic characteristics of the elastic and heterogeneous cloud resources. In this article, we develop a novel multiobjective optimization technique for workflow scheduling such that the makespan and cost can be minimized simultaneously. A scoring criterion, dynamic hierarchical structure and its evolutionary rules, and adaptive adjustment strategies are designed to cooperate with each other and increase the search ability and efficiency of the original and widely used NSGA-II. Adequate experiments are conducted to verify the proposed method&#x2019;s performance, and the experimental results show that it can provide more near-optimal solutions than the existing methods. It can be readily applied for implementing more efficient and effective cloud data centers to execute large-scale scientific workflows.

Reproducing GW150914: The First Observation of Gravitational Waves From a Binary Black Hole Merger

In 2016, LIGO and Virgo announced the first observation of gravitational waves from a binary black hole merger, known as GW150914. To establish the confidence of this detection, large-scale scientific workflows were used to measure the event's statistical significance. They used code written by the LIGO/Virgo and were executed on the LIGO Data Grid. The codes are publicly available, but there has not yet been an attempt to directly reproduce the results, although several analyses have replicated the analysis, confirming the detection. We attempt to reproduce the result presented in the GW150914 discovery paper using publicly available code on the Open Science Grid. We show that we can reproduce the main result but we cannot exactly reproduce the LIGO analysis as the original data set used is not public. We discuss the challenges we encountered and make recommendations for scientists who wish to make their work reproducible.

Cost-effective workflow scheduling approach on cloud under deadline constraint using firefly algorithm

Cloud computing, a novel and promising methodology in the distributed computing domain, provides a pay-per-use framework to solve large-scale scientific and business workflow applications. These workflow applications have a constraint that each of them must completed within the limited time (deadline constraint). Therefore, scheduling a workflow with deadline constraints is increasingly becoming a crucial research issue. However, many analytical reviews on scheduling problems reveal that existing solutions fail to provide cost-effective solutions and they do not consider the parameters like CPU performance variation, delay in acquisition and termination of Virtual Machines (VMs). This paper presents a Cost-Effective Firefly based Algorithm (CEFA) to solve workflow scheduling problems that can occur in an Infrastructure as a Service (IaaS) platform. The proposed CEFA uses a novel method for problem encoding, population initialization and fitness evaluation with an objective to provide cost-effective and optimized workflow execution within the time limit. The performance of the proposed CEFA is compared with the state-of-the-art algorithms such as IaaS Cloud-Partial Critical Path (IC-PCP), Particle Swarm Optimization (PSO), Robustness-Cost-Time (RCT), Robustness-Time-Cost (RTC), and Regressive Whale Optimization (RWO). Our experimental results demonstrate that the proposed CEFA outperforms current state-of-the-art heuristics with the criteria of achieving the deadline constraint and minimizing the cost of execution.

GMTA: A Geo-Aware Multi-Agent Task Allocation Approach for Scientific Workflows in Container-Based Cloud

Scientific workflow scheduling is one of the most challenging problems in cloud computing because of the large-scale computing tasks and massive data volumes involved. A cloud system is a distributed system that follows the on-demand resource provisioning and pay-per-use billing model. Therefore, practical scheduling approaches are essential for good workflow performance and low overheads. This paper proposes a novel workflow allocation approach, the Geo-aware Multiagent Task Allocation Approach (GMTA), which aims to optimize large-scale scientific workflow execution in container-based clouds. GMTA is an agent-based workflow allocation method that includes a market-like agent negotiation mechanism and a dynamic workflow restructuring strategy. It decreases workflow makespans and traffic overheads by reasonable task replications. Furthermore, the performance of GMTA is verified on real scientific workflows in the CloudSim environment.

Dynamic and semantic-aware access-control model for privacy preservation in multiple data center environments

With the rapid development of intelligent perception and other data acquisition technologies in the Internet of things, large-scale scientific workflows have been widely used in geographically distributed multiple data centers to realize high performance in business model construction and computational processing. However, insider threats pose very significant privacy and security risks to systems. Traditional access-control models can no longer satisfy the reasonable authorization of resources in these new cross-domain environments. Therefore, a dynamic and semantic-aware access-control model is proposed for privacy preservation in multiple data center environments, which implements a semantic dynamic authorization strategy based on an anomaly assessment of users’ behavior sequences. The experimental results demonstrate that this dynamic and semantic-aware access-control model is highly dynamic and flexible and can improve the security of the application system.

Scheduling large-scale scientific workflow on virtual machines with different numbers of vCPUs

With the wide deployment of cloud computing in scientific computing, cost minimization is increasingly critical for large-scale scientific workflow. Unfortunately, due to the highly intricate directed acyclic graph (DAG)-based workflow and the flexible usage of virtual machines (VMs) in cloud platform, the existing workflow scheduling approaches are inefficient to strike a balance between the parallelism and the topology of the DAG-based workflow while using the VMs, which causes a low utilization of VMs and consumes more cost. To address these issues, this paper presents a novel task scheduling framework named cost minimization approach with the DAG splitting method (COMSE) for minimizing the cost of running a deadline-constrained large-scale scientific workflow. First, we provide comprehensive theoretical analyses on how to improve the utilization of a resource-balanced multi-vCPU VM for running multiple tasks simultaneously. Second, considering the balance between the parallelism and the topology of a workflow, we simplify the DAG-based workflow, and based on the simplified DAG, a DAG splitting method is devised to preprocess the workflow. Third, since the cloud is charged by hours, we also design an exact algorithm to find the optimal operation pattern for a given schedule to make the consumed instance hours minimum, and this algorithm is named as instance hours minimization by Dijkstra (TOID). Finally, by employing the DAG splitting method and the TOID, the COMSE schedules a deadline-constrained large-scale scientific workflow on the multi-vCPU VMs and incorporates two important objects: minimizing the computation cost and the communication cost. Our solution approach is evaluated through rigorous performance evaluation study using real-word workflows, and the results show that the proposed COMSE approach outperforms existing algorithms in terms of computation cost and communication cost.

A Data and Task Co-Scheduling Algorithm for Scientific Cloud Workflows

Cloud computing has emerged as a promising computational infrastructure for cost-efficient workflow execution by provisioning on-demand resources in a pay-as-you-go manner. While scientific workflows require accessing community-wide resources, they usually need to be performed in collaborative cloud environments composed of multiple datacenters. Although such environments facilitate scientific collaboration, the movements of input and intermediate datasets across geographically distributed datacenters may cause intolerable latency that would hinder efficient execution of large-scale data-intensive scientific workflows. To address the problem, in this article we propose a novel multi-level K-cut graph partitioning algorithm to minimize the volume of data transfer across datacenters while satisfying load balancing and fixed data constraints. The algorithm first contracts the fixed input datasets in the same datacenter and their consuming tasks, and coarsens the contracted graph to a predefined scale in a level-by-level manner. Then, a K-cut algorithm is used to partition the resulted graph into K parts such that the cut size is minimized. After that, the partitioned graph is projected back to the original workflow graph, during which the load balancing constraint is maintained. We evaluate our algorithm using three real-world workflow applications and the results demonstrate that the proposed algorithm outperforms other state-of-the-art algorithms.

Budget aware scheduling algorithm for workflow applications in IaaS clouds

Cloud computing, a novel and promising model of Service-oriented computing, provides a pay-per-use framework to solve large-scale scientific and business workflow applications. Workflow scheduling in cloud is challenging due to dynamic nature of the cloud, particularly, on demand provisioning, elasticity, heterogeneous resource types, static & dynamic pricing models and virtualization. An example of workflow scheduling is mapping workflow tasks to cloud computing resources. Additionally, these workflow applications have a runtime constraint—the most typical being the cost of the computation and the time that computation requires to complete. Therefore, the focus is on two criteria: makespan and cost. This paper presents an algorithm called NBWS (Normalization based Budget constraint Workflow Scheduling) which generates a workflow schedule which minimizes the schedule length while satisfying the budget constraint. The algorithm undergoes a process of min–max normalization tailed by computing expect reasonable budget $$ (erb) $$ for dispatching the workflow tasks into one of the virtual machines. To minimize the execution time, NBWS algorithm maps the workflow tasks to resources which are having the earliest finish time within the allocated budget. The experimental results demonstrate that NBWS outperforms current state-of-the-art heuristics with respect to budget constraint and minimizing the makespan.

Detecting anomalous packets in network transfers: investigations using PCA, autoencoder and isolation forest in TCP

Large-scale scientific workflows rely heavily on high-performance file transfers. These transfers require strict quality parameters such as guaranteed bandwidth, no packet loss or data duplication. To have successful file transfers, methods such as predetermined thresholds and statistical analysis need to be done to determine abnormal patterns. Network administrators routinely monitor and analyze network data for diagnosing and alleviating these, making decisions based on their experience. However, as networks grow and become complex, monitoring large data files and quickly processing them, makes it improbable to identify errors and rectify these. Abnormal file transfers have been classified by simply setting alert thresholds, via tools such as PerfSonar and TCP statistics (Tstat). This paper investigates the feasibility of unsupervised feature extraction methods for identifying network anomaly patterns with three unsupervised classification methods—principal component analysis, autoencoder and isolation forest. We collect file transfer statistics from two experiment sets—synthetic iPerf generated traffic and 1000 Genome workflow runs, with synthetically introduced anomalies. Our results show that while PCA and a simple autoencoder finds it difficult to detect clusters, the tree-variant isolation forest is able to identify anomalous packets by breaking down TCP traces into tree classes early.

Adaptive Resource Allocation and Consolidation for Scientific Workflow Scheduling in Multi-Cloud Environments

The emerging multi-cloud environments (MCEs) empower the execution of large-scale scientific workflows (SWs) with sufficient resource provisioning. However, due to complex task dependencies in SWs and various cost-performance of cloud resources, the SW scheduling in MCEs faces huge challenges. To address these challenges, we propose an Online Workflow Scheduling algorithm based on Adaptive resource Allocation and Consolidation (OWS-A2C). In OWS-A2C, the deadline reassignment is first executed for SW tasks based on the execution performance of instance resources, which enhances resource utilization from a local perspective when executing an SW. Next, the execution instances are allocated and consolidated according to the performance requirements of multiple SWs, which improves resource utilization and reduces the total costs of executing multiple SWs from a global perspective. Finally, the SW tasks are dynamically scheduled to the execution instances with the earliest-deadline-first (EDF) discipline and completed before their sub-deadlines. The extensive simulation experiments are conducted to demonstrate the effectiveness of the proposed OWS-A2C on SW scheduling in MCEs, which outperforms three baseline scheduling methods with higher resource utilization and lower execution costs under deadline constraints.

Cost and makespan-aware workflow scheduling in hybrid clouds

Abstract Benefiting from rich resources and virtualization technologies, hybrid cloud has emerged as a promising solution to processing large-scale scientific workflow applications for users in a pay-as-you-go manner. However, considering the complexity of resource configuration and deployment in hybrid clouds, existing workflow scheduling strategies designed for traditional distributed computing systems are limited and powerless. Therefore, for profit-driven infrastructure-as-a-service (IaaS) cloud providers, minimizing makespan and monetary cost of scheduling scientific workflows is an imperative concern. In this paper, we propose two efficient workflow scheduling approaches for hybrid clouds that both consider makespan and monetary cost. Specifically, we first propose a single-objective workflow scheduling optimization approach called DCOH (deadline-constrained cost optimization for hybrid clouds) for minimizing the monetary cost of scheduling workflows under deadline constraint. Based on DCOH, we further propose a multi-objective workflow scheduling optimization approach called MOH (multi-objective optimization for hybrid clouds) for optimizing makespan and monetary cost of scheduling workflows simultaneously. Extensive simulation experiments have been conducted to validate the effectiveness of DCOH and MOH. Simulation results show that our DCOH approach can reduce up to 100.0% monetary cost for users as compared to the competing algorithms under the same deadline constraint and our MOH approach can achieve better cost-makespan trade-off solutions as compared to the competing algorithms.

Multi-Objective Scientific-Workflow Scheduling With Data Movement Awareness in Cloud

Due to serving several purposes simultaneously, running scientific workflows on dynamic environments such as cloud computing, has become multi-objective scheduling. Among these purposes, Cost and Makespan are probably the most two primitive objectives. Another critical factor in a large-scale scientific workflow is tremendous amount of data during execution. Therefore, this work also includes Data Movement as an additional objective as it has a major impact on network utilization and energy consumption in network equipment in cloud data center. In considering these three objectives, this work proposes a framework for scheduling solutions which combines a new nodes clustering technique in Directed Acyclic Graph (DAG) model known as Multilevel Dependent Node Clustering (MDNC) and the multi-objective optimization, Extreme Nondominated Sorting Genetic Algorithm-III (E-NSGA-III). E-NSGA-III is the recent extension of Nondominated Sorting Genetic Algorithm (NSGA-III). Five well-known scientific workflows, CyberShake, Epigenomics, LIGO, Montage, and SIPHT are selected as testbeds, while the commonly known Hypervolume is chosen as the performance metric. In this work, MDNC is also experimented with both NSGA-III. Comparison among three approaches, E-NSGA-III alone, E-NSGA-III with Peer-to-Peer clustering and E-NSGA-III with MDNC are carried out. The superiority of the proposed framework among them and its limitation are discussed.

DR-SWDF: A Dynamically Reconfigurable Framework for Scientific Workflows Deployment in the Cloud

Workflows management systems (WfMS) are aimed for designing, scheduling, executing, reusing, and sharing workflows in distributed environments like the Cloud computing. With the emergence of e-science workflows, which are used in different domains like astronomy, life science, and physics, to model and execute vast series of dependents functionalities and a large amount of manipulated data, the workflow management systems are required to provide customizable programming environments to ease the programming effort required by scientists to orchestrate a computational science experiment. A key issue for e-science WfMS is how to deal with the change of the execution environment constraints and the variability and confliction of end users and cloud providers objectives for the execution of the same workflow or sub-workflow. They have to customize their management processes to insure the adaptability of the execution environment to the scientific workflows specificities, especially when dealing with large-scale (data, computing, I/O)-intensive workflows. In this paper, we propose a dynamically re-configurable framework for the deployment of scientific workflows in the Cloud (called DR-SWDF) that allows customizing the workflow deployment process according to a set of objectives and constraints of end users or cloud providers defined differently for the tasks or partitions of the same workflow. The DR-SWDF framework offers a K-means based algorithm that allows dynamically clustering the input workflows or sub-workflows in order to identify the most convenient techniques or algorithms to be applied for their scheduling and deployment in the cloud. The simulations results run on three examples large-scale scientific workflows show that our proposed framework can achieve better results than the use of a generic purpose approach.

The future of scientific workflows

Today’s computational, experimental, and observational sciences rely on computations that involve many related tasks. The success of a scientific mission often hinges on the computer automation of these workflows. In April 2015, the US Department of Energy (DOE) invited a diverse group of domain and computer scientists from national laboratories supported by the Office of Science, the National Nuclear Security Administration, from industry, and from academia to review the workflow requirements of DOE’s science and national security missions, to assess the current state of the art in science workflows, to understand the impact of emerging extreme-scale computing systems on those workflows, and to develop requirements for automated workflow management in future and existing environments. This article is a summary of the opinions of over 50 leading researchers attending this workshop. We highlight use cases, computing systems, workflow needs and conclude by summarizing the remaining challenges this community sees that inhibit large-scale scientific workflows from becoming a mainstream tool for extreme-scale science.

Runtime prediction of parallel applications with workload-aware clustering

Traditionally, many science fields require great support for a massive workflow, which utilizes multiple cores simultaneously. In order to support such large-scale scientific workflows, high-capacity parallel systems such as supercomputers are widely used. To increase the utilization of these systems, most schedulers use backfilling policy based on user's estimated runtime. However, it is found to be extremely inaccurate because users overestimate their jobs. Therefore, in this paper, an efficient machine learning approach is present to predict the runtime of parallel application. The proposed method is divided into three phases. First is to analyze important feature of the history log data by factor analysis. Second is to carry out clustering for the parallel program based on the important features. Third is to build a prediction models by pattern similarity of parallel program log data and estimate runtime. In the experiments, we use workload logs on parallel systems (i.e., NASA-iPSC, LANL-CM5, SDSC-Par95, SDSC-Par96, and CTC-SP2) to evaluate the effectiveness of our approach. Comparing root-mean-square error with other techniques, experimental results show that the proposed method improves the accuracy up to 69.56%.

사용자 맞춤형 분산 컴퓨팅을 위한 컨테이너 기반 클러스터 관리 시스템

Several fields of science have traditionally demanded large-scale workflow support, which requires thousands of central processing unit (CPU) cores. In order to support such large-scale scientific workflows, large-capacity cluster systems such as supercomputers are widely used. However, as users require a diversity of software packages and configurations, a system administrator has some trouble in making a service environment in real time. In this paper, we present a container-based cluster management platform and introduce an implementation case to minimize performance reduction and dynamically provide a distributed computing environment desired by users. This paper offers the following contributions. First, a container-based virtualization technology is assimilated with a resource and job management system to expand applicability to support large-scale scientific workflows. Second, an implementation case in which docker and HTCondor are interlocked is introduced. Lastly, docker and native performance comparison results using two widely known benchmark tools and Monte-Carlo simulation implemented using various programming languages are presented.

Enabling scalable scientific workflow management in the Cloud

Cloud computing is gaining tremendous momentum in both academia and industry. In this context, we define the term “Cloud Workflow” as the specification, execution and provenance tracking of large-scale scientific workflows, as well as the management of data and computing resources to support the execution of large-scale scientific workflows in the Cloud. In this paper, we first analyze the gap between these two complementary technologies, and what it means to bring Clouds and workflows together. Then, we present the key challenges in supporting Cloud workflows, and present our reference framework for scientific workflow management in the Cloud. Last we present our experience in integrating a scientific workflow management system—Swift into the Cloud. We discuss the performance of cluster provisioning within the OpenNebula Cloud platform, the Eucalyptus Cloud platform and Amazon EC2, and we demonstrate the capability and efficiency of the integration using a NASA MODIS image processing workflow and the Montage image mosaic workflow.Note to practitioners. Scientific workflow management plays a very important role for scientific computing and application coordination, while Cloud computing offers scalability and resource on-demand. We devise autonomous methods to integrate scientific workflow management systems with Cloud platforms and also provision resources for large scale workflows, which can facilitate scientists to easily manage their workflows in the Cloud, and take advantage of large scale Cloud resources. There are a few integration options and many challenges in the process, and the experience we gain will help researchers in migrating their workflow management systems and workflow applications into the Cloud.