Workflow Execution Research Articles

DR-SWDF: A Dynamically Reconfigurable Framework for Scientific Workflows Deployment in the Cloud

Workflows management systems (WfMS) are aimed for designing, scheduling, executing, reusing, and sharing workflows in distributed environments like the Cloud computing. With the emergence of e-science workflows, which are used in different domains like astronomy, life science, and physics, to model and execute vast series of dependents functionalities and a large amount of manipulated data, the workflow management systems are required to provide customizable programming environments to ease the programming effort required by scientists to orchestrate a computational science experiment. A key issue for e-science WfMS is how to deal with the change of the execution environment constraints and the variability and confliction of end users and cloud providers objectives for the execution of the same workflow or sub-workflow. They have to customize their management processes to insure the adaptability of the execution environment to the scientific workflows specificities, especially when dealing with large-scale (data, computing, I/O)-intensive workflows. In this paper, we propose a dynamically re-configurable framework for the deployment of scientific workflows in the Cloud (called DR-SWDF) that allows customizing the workflow deployment process according to a set of objectives and constraints of end users or cloud providers defined differently for the tasks or partitions of the same workflow. The DR-SWDF framework offers a K-means based algorithm that allows dynamically clustering the input workflows or sub-workflows in order to identify the most convenient techniques or algorithms to be applied for their scheduling and deployment in the cloud. The simulations results run on three examples large-scale scientific workflows show that our proposed framework can achieve better results than the use of a generic purpose approach.

TIminer: NGS data mining pipeline for cancer immunology and immunotherapy.

SummaryRecently, a number of powerful computational tools for dissecting tumor-immune cell interactions from next-generation sequencing data have been developed. However, the assembly of analytical pipelines and execution of multi-step workflows are laborious and involve a large number of intermediate steps with many dependencies and parameter settings. Here we present TIminer, an easy-to-use computational pipeline for mining tumor-immune cell interactions from next-generation sequencing data. TIminer enables integrative immunogenomic analyses, including: human leukocyte antigens typing, neoantigen prediction, characterization of immune infiltrates and quantification of tumor immunogenicity.Availability and implementationTIminer is freely available at http://icbi.i-med.ac.at/software/timiner/timiner.shtml.Supplementary information Supplementary data are available at Bioinformatics online.

Energy management for multiple real-time workflows on cyber–physical cloud systems

Cyber–physical cloud systems (CPCS) are extensions of cyber–physical systems (CPS) that expand the cyber-part and distribute it on-device and in-cloud. CPCS are considered large-scale heterogeneous distributed cloud computing systems that support execution of multiple workflows. This study aims to reduce the energy consumption of multiple real-time workflows on CPCS and it contains two objectives: (1) maximizing the number of workflows that are completed within their deadlines; (2) minimizing the energy consumption of the workflows that are completed within their deadlines. The former is solved by proposing a deadline-driven processor merging for multiple workflows (DPMMW) algorithm, whereas the latter is solved by proposing a global energy saving for multiple workflows (GESMW) algorithm to minimize the total energy consumption. Experimental results validate that the combined DPMMW&GESMW algorithm can reduce deadline miss ratio (DMR) and save as much as possible energy over existing methods.

Taxonomy of workflow partitioning problems and methods in distributed environments

Abstract A workflow model is the computerized representation of a business or scientific process. It defines the starting and ending conditions of the process, the activities in the process, control flow and data flow among these activities, etc. A partitioning method creates workflow fragments that group some of the workflow model elements (activities, control flows, data flows). Workflow partitioning forms the foundation of decentralised workflow execution and increases scalability, and reuse of partitions. In the literature, different methods have been presented for workflow partitioning and offer a variety of execution approaches; however, there is no existing comprehensive survey and taxonomy of workflow partitioning methods. This article presents an overview of taxonomies characterizing the key concepts of the workflow life cycle and workflow partitioning methods through a comprehensive survey of business and scientific domains in decentralised environments. This in-depth analysis of taxonomies can provide researchers, designers and applications developers with clear guideline to compare current workflow partitioning methods to choose, reuse and compose more vigorous approaches. The article further presents research discussions and future challenges in this area .

Visual Simple Transformations

Empowering end-users to wire Internet of Things (IoT) objects (things and services) together would allow them to more easily conceive and realize interesting IoT solutions. A challenge lies in devising a simple end-user development approach to support the specification of transformations, which can bridge the mismatch in the data being exchanged among IoT objects. To tackle this challenge, we present Visual Simple Transformations (ViSiT) as an approach that allows end-users to use a jigsaw puzzle metaphor for specifying transformations that are automatically converted into underlying executable workflows. ViSiT is explained by presenting meta-models and an architecture for implementing a system of connected IoT objects. A tool is provided for supporting end-users in visually developing and testing transformations. Another tool is also provided for allowing software developers to modify, if they wish, a transformation's underlying implementation. This work was evaluated from a technical perspective by developing transformations and measuring ViSiT's efficiency and scalability and by constructing an example application to show ViSiT's practicality. A study was conducted to evaluate this work from an end-user perspective, and its results showed positive indications of perceived usability, learnability, and the ability to conceive real-life scenarios for ViSiT.

In‐memory staging and data‐centric task placement for coupled scientific simulation workflows

SummaryCoupled scientific simulation workflows are composed of heterogeneous component applications that simulate different aspects of the physical phenomena being modeled and that interact and exchange significant volumes of data at runtime. As the data volumes and generation rates keep growing, the traditional disk I/O–based data movement approach becomes cost prohibitive, and workflow requires more scalable and efficient approach to support the data movement. Moreover, the cost of moving large volume of data over system interconnection network becomes dominating and significantly impacts the workflow execution time. Minimize the amount of network data movement and localize data transfers are critical for reducing such cost. To achieve this, workflow task placement should exploit data locality to the extent possible and move computation closer to data. In this paper, we investigate applying in‐memory data staging and data‐centric task placement to reduce the data movement cost in large‐scale coupled simulation workflows. Specifically, we present a distributed data sharing and task execution framework that (1) co‐locates in‐memory data staging on application compute nodes to store data that needs to be shared or exchanged and (2) uses data‐centric task placement to map computations onto processor cores that a large portion of the data exchanges can be performed using the intra‐node shared memory. We also present the implementation of the framework and its experimental evaluation on Titan Cray XK7 petascale supercomputer.

Pallas

Software optimization is constantly a serious concern for developing high-performance systems. To accelerate the workflow execution of a specific functionality, software developers usually define and implement a fast path to speed up the critical and commonly executed functions in the workflow. However, producing a bug-free fast path is nontrivial. Our study on the Linux kernel discloses that a committed fast path can have up to 19 follow-up patches for bug fixing, and most of them are deep semantic bugs, which are difficult to be pinpointed by existing bug-finding tools. In this paper, we present such a new category of software bugs based on our fast-path bug study across various system software including virtual memory manager, file systems, network, and device drivers. We investigate their root causes and identify five error-prone aspects in a fast path: path state, trigger condition, path output, fault handling, and assistant data structure. We find that many of the deep bugs can be prevented by applying static analysis incorporating simple semantic information. We extract a set of rules based on our findings and build a toolkit PALLAS to check fast-path bugs. The evaluation results show that PALLAS can effectively reveal fast-path bugs in a variety of systems including Linux kernel, mobile operating system, software-defined networking system, and web browser.

Pallas

Software optimization is constantly a serious concern for developing high-performance systems. To accelerate the workflow execution of a specific functionality, software developers usually define and implement a fast path to speed up the critical and commonly executed functions in the workflow. However, producing a bug-free fast path is nontrivial. Our study on the Linux kernel discloses that a committed fast path can have up to 19 follow-up patches for bug fixing, and most of them are deep semantic bugs, which are difficult to be pinpointed by existing bug-finding tools. In this paper, we present such a new category of software bugs based on our fast-path bug study across various system software including virtual memory manager, file systems, network, and device drivers. We investigate their root causes and identify five error-prone aspects in a fast path: path state, trigger condition, path output, fault handling, and assistant data structure. We find that many of the deep bugs can be prevented by applying static analysis incorporating simple semantic information. We extract a set of rules based on our findings and build a toolkit PALLAS to check fast-path bugs. The evaluation results show that PALLAS can effectively reveal fast-path bugs in a variety of systems including Linux kernel, mobile operating system, software-defined networking system, and web browser.

Cümülön-D

We present a system called Cümülön-D for matrix-based data analysis in a spot market of a public cloud. Prices in such markets fluctuate over time: while users can acquire machines usually at a very low bid price, the cloud can terminate these machines as soon as the market price exceeds their bid price. The distinguishing features of Cümülön-D include its continuous, proactive adaptation to the changing market, and its ability to quantify and control the monetary risk involved in paying for a workflow execution. We solve the dynamic optimization problem in a principled manner with a Markov decision process, and account for practical details that are often ignored previously but nonetheless important to performance. We evaluate Cümülön-D's effectiveness and advantages over previous approaches with experiments on Amazon EC2.

Optimization of data-intensive workflows in stream-based data processing models

Stream computing applications require minimum latency and high throughput for efficiently processing real-time data. Typically, data-intensive applications where large datasets are required to be moved across execution nodes have low latency requirements. In this paper, a stream-based data processing model is adopted to develop an algorithm for optimal partitioning the input data such that the inter-partition data flow remains minimal. The proposed algorithm improves the execution of the data-intensive workflows in heterogeneous computing environments by partitioning the data-intensive workflow and mapping each partition on the available heterogeneous resources that offer minimum execution time. Minimum data movement between the partitions reduces the latency, which can be further reduced by applying advanced data parallelism techniques. In this paper, we apply data parallelism technique to the bottleneck (most compute-intensive) task in each partition that significantly reduces the latency. We study the effectiveness and the performance of the proposed approach by using synthesized workflows and real-world applications, such as Montage and Cybershake. Our evaluation shows that the proposed algorithm provides schedules with approximately 12% reduced latency and nearly 17% enhanced throughput as compared to the existing state of the art algorithms.

PSO-DS: a scheduling engine for scientific workflow managers

Cloud computing, an important source of computing power for the scientific community, requires enhanced tools for an efficient use of resources. Current solutions for workflows execution lack frameworks to deeply analyze applications and consider realistic execution times as well as computation costs. In this study, we propose cloud user---provider affiliation (CUPA) to guide workflow's owners in identifying the required tools to have his/her application running. Additionally, we develop PSO-DS, a specialized scheduling algorithm based on particle swarm optimization. CUPA encompasses the interaction of cloud resources, workflow manager system and scheduling algorithm. Its featured scheduler PSO-DS is capable of converging strategic tasks distribution among resources to efficiently optimize makespan and monetary cost. We compared PSO-DS performance against four well-known scientific workflow schedulers. In a test bed based on VMware vSphere, schedulers mapped five up-to-date benchmarks representing different scientific areas. PSO-DS proved its efficiency by reducing makespan and monetary cost of tested workflows by 75 and 78%, respectively, when compared with other algorithms. CUPA, with the featured PSO-DS, opens the path to develop a full system in which scientific cloud users can run their computationally expensive experiments.

A scalable deep learning platform for identifying geologic features from seismic attributes

The modern requirement for analyzing and interpreting ever-larger volumes of seismic data to identify prospective hydrocarbon prospects within stringent time deadlines represents an ongoing challenge in petroleum exploration. To provide a computer-based aid in addressing this challenge, we have developed a “big data” platform to facilitate the work of geophysicists in interpreting and analyzing large volumes of seismic data with scalable performance. We have constructed this platform on a modern distributed-memory infrastructure, providing a customized seismic analytics software development toolkit, and a Web-based graphical workflow interface along with a remote 3D visualization capability. These support the management of seismic data volumes, attributes processing, seismic analytics model development, workflow execution, and 3D volume visualization on a scalable, distributed computing platform. Early experiences show that computationally demanding deep learning methods such as convolutional neural networks (CNN) provide improved results over traditional methods such as support vector machines (SVMs) and logistic regression for identifying geologic faults in 3D seismic volumes. Our experiments show encouraging accuracy in identifying faults by combining CNN and traditional machine learning models with a variety of seismic attributes, and the platform is able to deliver scalable performance.

A cross-layer optimized storage system for workflow applications

This paper proposes using file system custom metadata as a bidirectional communication channel between applications and the storage middleware. This channel can be used to pass hints that enable cross-layer optimizations, an option hindered today by the ossified file-system interface. We study this approach in the context of storage system support for large-scale workflow execution systems: Our workflow-optimized storage system (WOSS), exploits application hints to provide per-file optimized operations, and exposes data location to enable location-aware scheduling. We argue that an incremental adoption path for adopting cross-layer optimizations in storage exists, present the system architecture for a workflow-optimized storage system and its integration with a workflow runtime engine, and evaluate this approach using synthetic and real applications over multiple success metrics (application runtime, generated network stress, and energy). Our performance evaluation demonstrates that this design brings sizeable performance gains. On a large scale cluster (100 nodes), compared to two production class distributed storage systems (Ceph and GlusterFS), WOSS achieves up to 6× better performance for the synthetic benchmarks and 20–40% better application-level performance gain for real applications.

A characterization of workflow management systems for extreme-scale applications

Automation of the execution of computational tasks is at the heart of improving scientific productivity. Over the last years, scientific workflows have been established as an important abstraction that captures data processing and computation of large and complex scientific applications. By allowing scientists to model and express entire data processing steps and their dependencies, workflow management systems relieve scientists from the details of an application and manage its execution on a computational infrastructure. As the resource requirements of today’s computational and data science applications that process vast amounts of data keep increasing, there is a compelling case for a new generation of advances in high-performance computing, commonly termed as extreme-scale computing, which will bring forth multiple challenges for the design of workflow applications and management systems. This paper presents a novel characterization of workflow management systems using features commonly associated with extreme-scale computing applications. We classify 15 popular workflow management systems in terms of workflow execution models, heterogeneous computing environments, and data access methods. The paper also surveys workflow applications and identifies gaps for future research on the road to extreme-scale workflows and management systems.

On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems

We study the problem of troubleshooting machine learning systems that rely on analytical pipelines of distinct components. Understanding and fixing errors that arise in such integrative systems is difficult as failures can occur at multiple points in the execution workflow. Moreover, errors can propagate, become amplified or be suppressed, making blame assignment difficult. We propose a human-in-the-loop methodology which leverages human intellect for troubleshooting system failures. The approach simulates potential component fixes through human computation tasks and measures the expected improvements in the holistic behavior of the system. The method provides guidance to designers about how they can best improve the system. We demonstrate the effectiveness of the approach on an automated image captioning system that has been pressed into real-world use.

Using ontologies for verification and validation of workflow-based experiments

Scientific experiments performed in the eScience domain require special tooling, software, and workflows that allow researchers to link, transform, visualise and interpret data. Recent studies report that such experiments often cannot be replicated due to differences in the underlying infrastructure. The provenance collection mechanisms were built into workflow engines to increase research replicability. However, the traces do not contain the execution context that consists of software, hardware and external services used to produce the result which may change between executions.The problem thus remains on how to identify such context and how to store such data. To address this challenge we propose the context model that integrates ontologies which describe workflow and its environment. It includes not only high level description of workflow steps and services but also low level technical details on infrastructure, including hardware, software, and files. In this paper we discuss which ontologies that compose the context model must be instantiated to enable verification of a workflow re-execution. We use a tool that monitors a workflow execution and automatically creates the context model. We also authored the VPlan ontology that enables modelling validation requirements. It contains a controlled vocabulary of metrics that can be used for quantification of requirements. We evaluate the proposed ontologies on five Taverna workflows that differ in the degree on which they depend on additional software and services.The results show that the proposed ontologies are necessary and can be used for verification and validation of scientific workflows re-executions in different environments without the necessity of accessing the original environment at the same time. Thus the scientists can state whether the scientific experiment is replicable.

Abstract, link, publish, exploit: An end to end framework for workflow sharing

Scientific workflows are increasingly used to manage and share scientific computations and methods to analyze data. A variety of systems have been developed that store the workflows executed and make them part of public repositories However, workflows are published in the idiosyncratic format of the workflow system used for the creation and execution of the workflows. Browsing, linking and using the stored workflows and their results often becomes a challenge for scientists who may only be familiar with one system. In this paper we present an approach for addressing this issue by publishing and exploiting workflows as data on the Web with a representation that is independent from the workflow system used to create them. In order to achieve our goal, we follow the Linked Data Principles to publish workflow inputs, intermediate results, outputs and codes; and we reuse and extend well established standards like W3C PROV. We illustrate our approach by publishing workflows and consuming them with different tools designed to address common scenarios for workflow exploitation.

Execution time estimation for workflow scheduling

Estimation of the execution time is an important part of the workflow scheduling problem. The aim of this paper is to highlight common problems in estimating the workflow execution time and propose a solution that takes into account the complexity and the stochastic aspects of the workflow components as well as their runtime. The solution proposed in this paper addresses the problems at different levels from a task to a workflow, including the error measurement and the theory behind the estimation algorithm. The proposed makespan estimation algorithm can be integrated easily into a wide class of schedulers as a separate module. We use a dual stochastic representation, characteristic/distribution function, in order to combine task estimates into the overall workflow makespan. Additionally, we propose the workflow reductions—operations on a workflow graph that do not decrease the accuracy of the estimates but simplify the graph structure, hence increasing the performance of the algorithm. Another very important feature of our work is that we integrate the described estimation schema into earlier developed scheduling algorithm GAHEFT and experimentally evaluate the performance of the enhanced solution in the real environment using the CLAVIRE platform.

Raw data queries during data-intensive parallel workflow execution

Computer simulations consume and produce huge amounts of raw data files presented in different formats, e.g., HDF5 in computational fluid dynamics simulations. Users often need to analyze domain-specific data based on related data elements from multiple files during the execution of computer simulations. In a raw data analysis, one should identify regions of interest in the data space and retrieve the content of specific related raw data files. Existing solutions, such as FastBit and RAW, are limited to a single raw data file analysis and can only be used after the execution of computer simulations. Scientific Workflow Management Systems (SWMS) can manage the dataflow of computer simulations and register related raw data files at a provenance database. This paper aims to combine the advantages of a dataflow-aware SWMS and the raw data file analysis techniques to allow for queries on raw data file elements that are related, but reside in separate files. We propose a component-based architecture, named as ARMFUL (Analysis of Raw data from Multiple Files) with raw data extraction and indexing techniques, which allows for a direct access to specific elements or regions of raw data space. ARMFUL innovates by using a SWMS provenance database to add a dataflow access path to raw data files. ARMFUL facilitates the invocation of ad-hoc programs and third party tools (e.g., FastBit tool) for raw data analyses. In our experiments, a real parallel computational fluid dynamics is executed, exploring different alternatives of raw data extraction, indexing and analysis.

An efficient temporal verification algorithm for intersectant constraints in scientific workflow

It is usually essential to conduct temporal verification in order to ensure overall temporal correctness for the usefulness of execution results in a scientific workflow. For improving the efficiency of temporal verification, many efforts have been dedicated to the selection of more effective and efficient checkpoints along scientific workflow execution so that temporal violations can be found and handled in time. Most of checkpoint selection strategies involve temporal constraints, but the relations among the temporal constraints have been ignored. The only existing strategy considering relations among temporal constraints is for the nested situation and temporal dependency based only. However, the intersectant situation and non-dependency relationship of temporal constraints have to be considered. In this paper, a constraint adjustment strategy-based algorithm for efficient temporal verification in scientific workflow involved in both temporal dependency and temporal reverse-dependency is presented for the intersectant situation. Simulations show that the temporal verification efficiency of our algorithm is significantly improved.