Data-intensive Workflows Research Articles

Measuring the impact of burst buffers on data-intensive scientific workflows

Science applications frequently produce and consume large volumes of data, but delivering this data to and from compute resources can be challenging, as parallel file system performance is not keeping up with compute and memory performance. To mitigate this I/O bottleneck, some systems have deployed burst buffers, but their impact on performance for real-world scientific workflow applications is still not clear. In this paper, we examine the impact of burst buffers through the remote-shared, allocatable burst buffers on the Cori system at NERSC. By running two data-intensive workflows, a high-throughput genome analysis workflow, and a subset of the SCEC high-performance CyberShake workflow, a production seismic hazard analysis workflow, we find that using burst buffers offers read and write improvements of an order of magnitude, and these improvements lead to increased job performance, and thereby increased overall workflow performance, even for long-running CPU-bound jobs.

Data-Intensive Workflow Management: For Clouds and Data-Intensive and Scalable Computing Environments

Abstract Workflows may be defined as abstractions used to model the coherent flow of activities in the context of an in silico scientific experiment. They are employed in many domains of science su...

A cost-based storage format selector for materialized results in big data frameworks

Modern big data frameworks (such as Hadoop and Spark) allow multiple users to do large-scale analysis simultaneously, by deploying data-intensive workflows (DIWs). These DIWs of different users share many common tasks (i.e, 50–80%), which can be materialized and reused in future executions. Materializing the output of such common tasks improves the overall processing time of DIWs and also saves computational resources. Current solutions for materialization store data on Distributed File Systems by using a fixed storage format. However, a fixed choice is not the optimal one for every situation. Specifically, different layouts (i.e., horizontal, vertical or hybrid) have a huge impact on execution, according to the access patterns of the subsequent operations. In this paper, we present a cost-based approach that helps deciding the most appropriate storage format in every situation. A generic cost-based framework that selects the best format by considering the three main layouts is presented. Then, we use our framework to instantiate cost models for specific Hadoop storage formats (namely SequenceFile, Avro and Parquet), and test it with two standard benchmark suits. Our solution gives on average 1.33$$\times $$ speedup over fixed SequenceFile, 1.11$$\times $$ speedup over fixed Avro, 1.32$$\times $$ speedup over fixed Parquet, and overall, it provides 1.25$$\times $$ speedup.

A Dynamic Data-throttling Approach to Minimize Workflow Imbalance

Scientific workflows enable scientists to undertake analysis on large datasets and perform complex scientific simulations. These workflows are often mapped onto distributed and parallel computational infrastructures to speed up their executions. Prior to its execution, a workflow structure may suffer transformations to accommodate the computing infrastructures, normally involving task clustering and partitioning. However, these transformations may cause workflow imbalance because of the difference between execution task times (runtime imbalance) or because of unconsidered data dependencies that lead to data locality issues (data imbalance). In this article, to mitigate these imbalances, we enhance the workflow lifecycle process in use by introducing a workflow imbalance phase that quantifies workflow imbalance after the transformations. Our technique is based on structural analysis of Petri nets, obtained by model transformation of a data-intensive workflow, and Linear Programming techniques. Our analysis can be used to assist workflow practitioners in finding more efficient ways of transforming and scheduling their workflows. Moreover, based on our analysis, we also propose a technique to mitigate workflow imbalance by data throttling. Our approach is based on autonomic computing principles that determine how data transmission must be throttled throughout workflow jobs. Our autonomic data-throttling approach mainly monitors the execution of the workflow and recompute data-throttling values when certain watchpoints are reached and time derivation is observed. We validate our approach by a formal proof and by simulations along with the Montage workflow. Our findings show that a dynamic data-throttling approach is feasible, does not introduce a significant overhead, and minimizes the usage of input buffers and network bandwidth.

Reinforcement Learning based scheduling in a workflow management system

Any computational process from simple data analytics tasks to training a machine learning model can be described by a workflow. Many workflow management systems (WMS) exist that undertake the task of scheduling workflows across distributed computational resources. In this work, we introduce a WMS that leverages machine learning to predict workflow task runtime and the probability of failure of task assignments to execution sites. The expected runtime of workflow tasks can be used to approximate the weight of the workflow graph branches in respect to the total workflow workload and the ability to anticipate task failures can discourage task assignments that are unlikely to succeed. We demonstrate that the proposed machine learning models can lead to significantly more informed scheduling decisions that minimize task failures and utilize execution sites more efficiently, thus leading to reduced workflow runtime. Additionally, we train a modified sequence-to-sequence neural network architecture via reinforcement learning to perform scheduling decisions as part of a WMS. Our approach introduces a WMS that can drastically improve its scheduling performance by independently learning over time, without external intervention or reliance on any specific heuristic or optimization technique. Finally, we test our approach in real-world scenarios utilizing computationally demanding and data intensive workflows and evaluate its performance against existing scheduling methodologies traditionally used in WMSes. The performance evaluation outcome confirms that the proposed approach significantly outperforms the other scheduling algorithms in a consistent manner and achieves the best execution runtime with the lowest number of failed tasks and communication costs.

Using simple PID-inspired controllers for online resilient resource management of distributed scientific workflows

Scientific workflows have become mainstream for conducting large-scale scientific research. As a result, many workflow applications and Workflow Management Systems (WMSs) have been developed as part of the cyberinfrastructure to allow scientists to execute their applications seamlessly on a range of distributed platforms. Although the scientific community has addressed this challenge from both theoretical and practical approaches, failure prediction, detection, and recovery still raise many research questions. In this paper, we propose an approach inspired by the control theory developed as part of autonomic computing to predict failures before they happen, and mitigated them when possible. The proposed approach is inspired on the proportional–integral–derivative controller (PID controller) control loop mechanism, which is widely used in industrial control systems, where the controller will react to adjust its output to mitigate faults. PID controllers aim to detect the possibility of a non-steady state far enough in advance so that an action can be performed to prevent it from happening. To demonstrate the feasibility of the approach, we tackle two common execution faults of large scale data-intensive workflows—data storage overload and memory overflow. We developed a simulator, which implements and evaluates simple standalone PID-inspired controllers to autonomously manage data and memory usage of a data-intensive bioinformatics workflow that consumes/produces over 4.4 TB of data, and requires over 24 TB of memory to run all tasks concurrently. Experimental results obtained via simulation indicate that workflow executions may significantly benefit from the controller-inspired approach, in particular under online and unknown conditions. Simulation results show that nearly-optimal executions (slowdown of 1.01) can be attained when using our proposed method, and faults are detected and mitigated far in advance of their occurrence.

Cost-effective replication management and scheduling in edge computing

The high volumes of data are continuously generated from Internet of Things (IoT) sensors in an industrial landscape. Especially, the data-intensive workflows from IoT systems require to be processed in a real-time, reliable and low-cost way. Edge computing can provide a low-latency and cost-effective computing paradigm to deploy workflows. Therefore, data replication management and scheduling for delay-sensitive workflows in edge computing have become challenge research issues. In this work, first, we propose a replication management system which includes dynamic replication creator, a specialized cost-effective scheduler for data placement, a system watcher and some data security tools for collaborative edge and cloud computing systems. And then, considering task dependency, data reliability and sharing, the data scheduling for the workflows is modeled as an integer programming problem. And we present the faster meta-heuristic algorithm to solve it. The experimental results show that our algorithms can achieve much better system performance than comparative traditional strategies, and they can create a suitable number of data copies and search the higher quality replica placement solution while reducing the total data access costs under the deadline constraint.

A novel scheduling algorithm for data-intensive workflow in virtualised clouds

In cloud platforms, workflow applications have been widely used to solve the complicated problems, which often need to process a large volume of data. However, the characteristic of data-intensive for these applications are easily to result in low execution efficiency due to networking traffic or congestion. In this paper, we present a workflow scheduling algorithm which is capable of minimising the cost of networking communication and therefore improving the execution efficiency of those workflow applications. Extensive experiments are conducted on some real-world workflows to examine the performance of the proposed algorithm, and the results show that it can significantly reduce the communication cost and improve the execution efficiency of data-intensive workflows comparing with existing algorithms.

Data-Aware Scheduling Strategy for Scientific Workflow Applications in IaaS Cloud Computing

Scientific workflows benefit from the cloud computing paradigm, which offers access to virtual resources provisioned on pay-as-you-go and on-demand basis. Minimizing resources costs to meet user’s budget is very important in a cloud environment. Several optimization approaches have been proposed to improve the performance and the cost of data-intensive scientific Workflow Scheduling (DiSWS) in cloud computing. However, in the literature, the majority of the DiSWS approaches focused on the use of heuristic and metaheuristic as an optimization method. Furthermore, the tasks hierarchy in data-intensive scientific workflows has not been extensively explored in the current literature. Specifically, in this paper, a data-intensive scientific workflow is represented as a hierarchy, which specifies hierarchical relations between workflow tasks, and an approach for data-intensive workflow scheduling applications is proposed. In this approach, first, the datasets and workflow tasks are modeled as a conditional probability matrix (CPM). Second, several data transformation and hierarchical clustering are applied to the CPM structure to determine the minimum number of virtual machines needed for the workflow execution. In this approach, the hierarchical clustering is done with respect to the budget imposed by the user. After data transformation and hierarchical clustering, the amount of data transmitted between clusters can be reduced, which can improve cost and makespan of the workflow by optimizing the use of virtual resources and network bandwidth. The performance and cost are analyzed using an extension of Cloudsim simulation tool and compared with existing multi-objective approaches. The results demonstrate that our approach reduces resources cost with respect to the user budgets.

A novel scheduling algorithm for data-intensive workflow in virtualised clouds

In cloud platforms, workflow applications have been widely used to solve the complicated problems, which often need to process a large volume of data. However, the characteristic of data-intensive for these applications are easily to result in low execution efficiency due to networking traffic or congestion. In this paper, we present a workflow scheduling algorithm which is capable of minimising the cost of networking communication and therefore improving the execution efficiency of those workflow applications. Extensive experiments are conducted on some real-world workflows to examine the performance of the proposed algorithm, and the results show that it can significantly reduce the communication cost and improve the execution efficiency of data-intensive workflows comparing with existing algorithms.

A novel data aware task clustering mechanism for scientific workflow applications in cloud

Scientific applications modelled as directed acyclic graphs (DAGs) are composed of complex calculations and a large amount of data transfer. It is very difficult to execute these applications on traditional distributed computing platforms. For such applications, cloud provides a reliable solution due to its unique characteristics, task clustering is performed which combines two or more tasks into a single executable unit. Task clustering can help to reduce the system overheads such as queue delay, engine delay and so on. Existing clustering algorithms in this domain focus more on computational granularity of the tasks without considering the data dependency among the tasks. In this paper, a data aware clustering algorithm has been proposed which combines the tasks depending on the size of data transferred between interdependent tasks. Experiments were conducted to compare the proposed clustering algorithm with the existing baseline and balanced clustering algorithms and it was observed that proposed algorithm gave better makespan and cost for data intensive workflow applications.

A novel data aware task clustering mechanism for scientific workflow applications in cloud

Scientific applications modelled as directed acyclic graphs (DAGs) are composed of complex calculations and a large amount of data transfer. It is very difficult to execute these applications on traditional distributed computing platforms. For such applications, cloud provides a reliable solution due to its unique characteristics, task clustering is performed which combines two or more tasks into a single executable unit. Task clustering can help to reduce the system overheads such as queue delay, engine delay and so on. Existing clustering algorithms in this domain focus more on computational granularity of the tasks without considering the data dependency among the tasks. In this paper, a data aware clustering algorithm has been proposed which combines the tasks depending on the size of data transferred between interdependent tasks. Experiments were conducted to compare the proposed clustering algorithm with the existing baseline and balanced clustering algorithms and it was observed that proposed algorithm gave better makespan and cost for data intensive workflow applications.

MOWS: Multi-objective workflow scheduling in cloud computing based on heuristic algorithm

Cloud computing is emerging with growing popularity in workflow scheduling, especially for scientific workflow. Deploying data-intensive workflows in the cloud brings new factors to be considered during specification and scheduling. Failure to establish intermediate data security may cause information leakage or data alteration in the cloud environment. Existing scheduling algorithms for the cloud disregard the interaction among tasks and its effects on application security requirements. To address this issue, we design a new systematic method that considers both tasks security demands and interactions in secure tasks placement in the cloud. In order to respect security and performance, we formulate a model for task scheduling and propose a heuristic algorithm which is based on task’s completion time and security requirements. In addition, we present a new attack response approach to reduce certain security threats in the cloud. To do so, we introduce task security sensitivity measurement to quantify tasks security requirements. We conduct extensive experiments to quantitatively evaluate the performance of our approach, using WorkflowSim, a well-known cloud simulation tool. Experimental results based on real-world workflows show that compared with existing algorithms, our proposed solution can improved the overall system security in terms of quality of security and security risk under a wide range of workload characteristics. Additionally, our results demonstrate that the proposed attack response algorithm can effectively reduce cloud environment threats.

Improving the energy efficiency and performance of data-intensive workflows in virtualized clouds

In recent years, deploying and running data-intensive workflows in cloud platform has become more and more popular in many areas. Unlike computation-intensive applications, a data-intensive workflow typically requires to deal with bulk data transferring between different resource sites, which means some traditional energy-efficiency optimization technologies are difficult to be enforced when running data-intensive workflows. In this paper, we first formulate the power model of a data-intensive workflow, which takes into account power consumption caused by data transferring. Based on this power model, we introduce a novel metric called Shortest Path in terms of Energy Consumption and design an energy-efficient heuristic scheduling algorithm, which is aiming at reducing the extra energy consumption caused by delays of bulk data transferring. Extensive experiments and performance evaluations show that the proposed scheduling algorithm can significantly reduce the overall energy consumption of running data-intensive workflows comparing with several existing algorithms. In addition, the proposed algorithm also exhibits better adaptiveness and robustness when a cloud system is facing intensive and unpredicted workloads.

Designing hybrid graph model and algorithmic analysis of workflow decomposition in mobile distributed systems

The execution of data intensive workflow is challenging in the domain of mobile distributed systems due to the involvement of heterogeneous computing devices and dynamic network topology. Moreover, data intensive workflow execution requires frequent access to database servers for dataflow and control flow operations resulting in massive data transfer. The workflow decomposition and distribution are required to minimize data transfer and to reduce execution delay in mobile distributed systems. This paper proposes a hybrid graph model of workflow in mobile distributed systems relying on dynamic network topology and distributed database in order to facilitate workflow decomposition. The algebraic and numerical analysis of the hybrid graph model is represented providing inherent properties. Furthermore, we have presented an extensive analytical study of existing workflow decomposition approaches and have provided detailed analysis to determine the suitability of these approaches in mobile distributed systems. We have also proposed a novel framework and implementation directions for workflow decomposition based on multiple parameters for mobile distributed systems.

Report from the Fourth Workshop on Algorithms andSystems for MapReduce and Beyond (BeyondMR '17)

This report summarizes the presentations and discussions of the fourth workshop on Algorithms and Systems for MapReduce and Beyond (BeyondMR '17). The BeyondMR workshop was held in conjunction with the 2017 SIGMOD/PODS conference in Chicago, Illinois, USA on Friday May 19, 2017. The goal of the workshop was to bring together researchers and practitioners to explore algorithms, computational models, languages and interfaces for systems that provide large-scale parallelization and fault tolerance. These include specialized programming and data-management systems based on MapReduce and extensions thereof, graph processing systems and data-intensive workflow systems. The program featured two well-attended invited talks, the first on current and future development in big data processing by Matei Zaharia from Databricks and the University of Stanford, and the second on computational models for the analysis and development of big data processing algorithms by Ke Yi from the Hong Kong University of Science and Technology.

An improved task scheduling algorithm for scientific workflow in cloud computing environment

As an emerging business computing model, cloud computing needs to deal with the scientific workflow submitted by user groups. How to efficiently schedule massive tasks of scientific workflow is an important problem in cloud computing. In order to minimize the total execution time of workflow, reduce the consume of cloud resources, reduce execution costs of users, a new task scheduling algorithm based on task duplication and task grouping is proposed in this paper. The new algorithm is composed of four steps. Firstly, the join nodes are duplicated, a DAG is converted into an in-tree graph, then all tasks are divide into task groups, it reduces communication overhead between tasks; then some task groups are merged by utilizing the idle time between tasks in a task group, it reduces the use of the processors; lastly, Assign the tasks to processors by making full use of the idle time of the processors, it increases resource utilization. The new algorithm is compared with TDS and TDCS by simulation platform CloudSim. The performance indicators for comparison include makespan of workflow, the number of used processors and resource utilization. The experiment results show that the new algorithm has a smaller makespan of workflow, fewer processors are used, and has higher resource utilization for both compute-intensive and data-intensive workflow, especially for data-intensive workflow, the new algorithm has obvious advantages on the three performance indicators.

Combining Static and Dynamic Storage Management for Data Intensive Scientific Workflows

Workflow management systems are widely used to express and execute highly parallel applications. For data-intensive workflows, storage can be the constraining resource: The number of tasks running at once must be artificially limited to not overflow the space available in the filesystem. It is all too easy for a user to dispatch a workflow which consumes all available storage and disrupts all system users. To address these issues, we present a three-tiered approach to workflow storage management: (1) A static analysis algorithm which analyzes the storage needs of a workflow before execution, giving a realistic prediction of success or failure. (2) An online storage management algorithm which accounts for the storage needed by future tasks to avoid deadlock at runtime. (3) A task containment system which limits storage consumption of individual tasks, enabling the strong guarantees of the static analysis and dynamic management algorithms. We demonstrate the application of these techniques on three complex workflows.

Supporting Data-Intensive Workflows in Software-Defined Federated Multi-Clouds

Cloud computing is emerging as a viable platform for scientific exploration. Elastic and on-demand access to resources (and other services), the abstraction of “unlimited” resources, and attractive pricing models provide incentives for scientists to move their workflows into clouds. Generalizing these concepts beyond a single virtualized datacenter, it is possible to create federated marketplaces where different types of resources (e.g., clouds, HPC grids, supercomputers) that may be geographically distributed, are collectively exposed as a single elastic infrastructure. This presents opportunities for optimizing the execution of application workflows with heterogeneous and dynamic requirements, and tackling larger scale problems. In this paper, we introduce a framework to manage the end-to-end execution of data-intensive application workflows in dynamic software-defined resource federation. This framework enables the autonomic execution of workflows by elastically provisioning an appropriate set of resources that meet application requirements, and by adapting this set of resources at runtime as the requirements change. It also allows users to customize scheduling policies that drive the way resources federated and used. To demonstrate the benefits of our approach, we study the execution of two different data-intensive scientific workflows in a multi-cloud federation using different policies and objective functions.

Data intensive ATLAS workflows in the Cloud

This contribution reports on the feasibility of executing data intensive workflows on Cloud infrastructures. In order to assess this, the metric ETC = Events/Time/Cost is formed, which quantifies the different workflow and infrastructure configurations that are tested against each other. In these tests ATLAS reconstruction Jobs are run, examining the effects of overcommitting (more parallel processes running than CPU cores available), scheduling (staggered execution) and scaling (number of cores). The desirability of commissioning storage in the Cloud is evaluated, in conjunction with a simple analytical model of the system, and correlated with questions about the network bandwidth, caches and what kind of storage to utilise. In the end a cost/benefit evaluation of different infrastructure configurations and workflows is undertaken, with the goal to find the maximum of the ETC value.