Workflow Graph Research Articles

Building a business process model using temporal knowledge when implementing process management

The article’s subject matter are business processes, which are a sequence of work, which are provided with resources and create products and services that are valuable for the customers of such processes. The aim is to develop an approach to solving the problems of implementing process management based on the automated detection of temporal knowledge, which makes it possible to establish conditions and restrictions on the execution of process actions, considering their actual temporal order, and thus makes it possible to consider the personal knowledge of process performers both in building and refining their model. Objectives: To structure the process management tasks, considering the interaction between the subtasks of process management implementation and business process management tasks; to develop a method for building a business process prototype using temporal knowledge. The approaches used are process management methods, process mining methods, temporal knowledge methods. The scientific novelty of the obtained results is as follows. The article develops a method for building a model of the business process "as it is", performed in an enterprise, using temporal knowledge. The method includes stages of forming temporal knowledge in the form of rules that determine the sequence of process actions in time, selection of subsets of rules that specify sequential and parallel or alternative execution of process actions, and stages of building the business process model in the form of a workflow graph that considers the identified temporal rules. In practice, the method creates conditions for the implementation of continuous improvement of the business process based on the iterative identification and further use of the tacit knowledge of the performers, which is reflected in the temporal rules.

Iterative Probabilistic Performance Prediction for Multiple IoT Applications in Contention

Internet of Things (IoT) has become omnipresent in many applications, such as healthcare, vehicles, and precision farming. They sense data from dozens of sensors scheduled periodically in a synchronous fashion on mobile CPUs that are forwarded to the cloud or other IoT devices via an essentially stochastic wireless channel. Hence, the task response time becomes stochastic, preventing optimization at compile time. On the other hand, knowing response time at compile time along with jitter, availability, and scalability is crucial to ensure a certain level of Quality of Service. This contribution presents a stochastic framework for performance analyses of multiapplications on a possible multiprocessor platform. When annotated with (stochastic) execution time, a traditional synchronous dataflow (SDF) graph can be transformed into a directed acyclic workflow graph, revealing the timing of individual actors. A generalized version of the rejection sampling Monte Carlo algorithm explores the properties of the workflow graph, to determine the distribution of the response time in a single application as well as a multiapplication multiple access scenario. Mean and jitter are the moments of the distribution. An IoT toy example with a number of distributed smart sensors was deployed in real environments to assess the performance of the proposed framework. Our analysis framework works at compile time of the code, scales with the number of things, and has low computational complexity.

Workflow performance prediction based on graph structure aware deep attention neural network

• In workflow performance prediction, DAG structure matters; • DAG-Transformer effectively embeds the DAG information and outperforms mainstream ML, DL , and GCN methods; • A new dataset for cloud workflow performance prediction is accompanied as well as the source code. With the rapid growth of cloud computing, efficient operational optimization and resource scheduling of complex cloud business processes rely on real-time and accurate performance prediction. Previous research on cloud computing performance prediction focused on qualitative (heuristic rules), model-driven, or coarse-grained time-series prediction, which ignore the study of historical performance, resource allocation status and service sequence relationships of workflow services. There are even fewer studies on prediction for workflow graph data due to the lack of available public datasets. In this study, from Alibaba Cloud's Cluster-trace-v2018, we extract nearly one billion offline task instance records into a new dataset, which contains approximately one million workflows and their corresponding directed acyclic graph (DAG) matrices. We propose a novel workflow performance prediction model (DAG-Transformer) to address the aforementioned challenges. In DAG-Transformer, we design a customized position encoding matrix and an attention mask for workflows, which can make full use of workflow sequential and graph relations to improve the embedding representation and perception ability of the deep neural network. The experiments validate the necessity of integrating graph-structure information in workflow prediction. Compared with mainstream deep learning (DL) methods and several classic machine learning (ML) algorithms, the accuracy of DAG-Transformer is the highest. DAG-Transformer can achieve 85-92% CPU prediction accuracy and 94-98% memory prediction accuracy, while maintaining high efficiency and low overheads. This study establishes a new paradigm and baseline for workflow performance prediction and provides a new way for facilitating workflow scheduling.

Distributed scheduling method for multiple workflows with parallelism prediction and DAG prioritizing for time constrained cloud applications

Fog computing is an emerging popular paradigm that extends the availability of resources to the network's edge in order to improve the quality metrics of existing Cloud-based applications. However, scheduling workflow applications with time-constraints are complex regarding the count of resources, physical topology of clusters, and the structure of the task graph of the workflows. Adding Fog resources to the intricate problem space of Cloud-based scheduling needs even more time-consuming and complicated algorithms. In this paper, a multi-criteria Mamdani fuzzy algorithm is proposed to analyze the workflow graphs with the assistance of a Long-Short Term Memory neural network parallelism prediction module. The group-based priority assignment schema performed by the fuzzy inference system assigns a priority value to workflows to indicate the relative precedence of requests. Distributed schedulers then send the workflows to target sites according to their current workloads. The whole process is performed in a decentralized manner to prevent any bottlenecks. We have used an extensive software simulation study to compare the proposed algorithm in real workloads with two recent and notable algorithms. The simulation results confirm the proposed algorithm's superiority in fulfilling time-constraints, resource utilization, and overall application scheduling success rate.

StreamFlow: Cross-Breeding Cloud With HPC

Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">entire</i> workflow in different environments, e.g., Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments, and that makes it possible the execution onto multiple sites not sharing a common data space. StreamFlow is then exemplified on a novel bioinformatics pipeline for single-cell transcriptomic data analysis workflow.

Using the Context-Sensitive Policy Mechanism for Building Data Acquisition Systems in Large Scale Distributed Cyber-Physical Systems Built on Fog Computing Platforms

The article deals with the use of context-sensitive policies in the building of data acquisition systems in large scale distributed cyber-physical systems built on fog computing platforms. It is pointed out that the distinctive features of modern cyber-physical systems are their high complexity and constantly changing structure and behavior, which complicates the data acquisition procedure. To solve this problem, it is proposed to use an approach according to which the data acquisition procedure is divided into two phases: model construction and data acquisition, which allows parallel realization of these procedures. A distinctive feature of the developed approach is that the models are built in runtime automatically. As a top-level model, a multi-level relative finite state operational automaton is used. The automaton state is described using a multi-level structural-behavioral model, which is a superposition of four graphs: the workflow graph, the data flow graph, the request flow graph and the resource graph. To implement the data acquisition procedure using the model, the context-sensitive policy mechanism is used. The article discusses possible approaches to implementation of suggested mechanisms and describes an example of application.

Executing cyclic scientific workflows in the cloud

We present an algorithm and a software architecture for a cloud-based system that executes cyclic scientific workflows whose structure may change during run time. Existing approaches either rely on workflow definitions based on directed acyclic graphs (DAGs) or require workarounds to implement cyclic structures. In contrast, our system supports cycles natively, avoids workarounds, and as such reduces the complexity of workflow modelling and maintenance. Our algorithm traverses workflow graphs and transforms them iteratively into linear sequences of executable actions. We call these sequences process chains. Our software architecture distributes the process chains to multiple compute nodes in the cloud and oversees their execution. We evaluate our approach by applying it to two practical use cases from the domains of astronomy and engineering. We also compare it with two existing workflow management systems. The evaluation demonstrates that our algorithm is able to execute dynamically changing workflows with cycles and that design and maintenance of complex workflows is easier than with existing solutions. It also shows that our software architecture can run process chains on multiple compute nodes in parallel to significantly speed up the workflow execution. An implementation of our algorithm and the software architecture is available with the Steep Workflow Management System that we released under an open-source license. The resources for the first practical use case are also available as open source for reproduction.

Bio-Inspired Workflow Scheduling on HPC Platforms

Efficient scheduling of tasks in workflows of cloud or grid applications is a key to achieving better utilization of resources as well as timely completion of the user jobs. Many scientific applications comprise several tasks that are dependent in nature and are specified by workflow graphs. The aim of the cloud meta-scheduler is to schedule the user application tasks (and the applications) so as to optimize the resource utilization and to execute the user applications in minimum amount of time. During the past decade, there have been several attempts to use bio-inspired scheduling algorithms to obtain an optimal or near optimal schedule in order to minimize the overall schedule length and to optimize the use of resources. However, as the number of tasks increases, the solution space comprising different tasks-resource mapping sequences increases exponentially. Hence, there is a need to devise mechanisms to improvise the search strategies of the bio-inspired scheduling algorithms for better scheduling solutions in lesser number of iterations/time. The objective of the research work in this paper is to use bio-inspired bacteria foraging optimization algorithm (BFOA) along with other heuristics algorithms for better search of the scheduling solution space for multiple workflows. The idea is to first find a schedule by the heuristic algorithms such as MaxMin, MinMin, and Myopic, and use these as initial solutions (along with other randomly generated solutions) in the search space to get better solutions using BFOA. The performance of our approach with the existing approaches is compared for quality of the scheduling solutions. The results demonstrate that our hybrid approach (MinMin/Myopic with BFOA) outperforms other approaches.

Unraveling Iterative Control Structures from Business Processes

Iterative control structures allow the repeated execution of tasks, activities or sub-processes according to the given conditions in a process model. Iterative control structures can significantly increase the risk of triggering temporal exceptions since activities within the scope of these control structures could be repeatedly executed until a predefined condition is met. In this paper, we propose two approaches to unravel iterative control structures from process models. The first approach unravels loops based on zero-one principle. The second approach unravels loops based on branching probabilities assigned at split gateways. The proposed methods can be used to unfold structured loops, nested loops and crossing loops. Since the unfolded model does not contain any iterative control structures, it can be used for further analysis by process designers during the modeling phase. The proposed methods are implemented based on workflow graphs, and therefore they are compatible with modeling languages such as Business Process Modelling Notation (BPMN). In the experiments, the execution behavior of unfolded process models is compared against the original models based on the concept of runs. Experimental results reveal that runs generated from the original models can be correctly executed in the unfolded BPMN models that do not contain any loops.

A Hybrid Approach for Detecting Prerequisite Relations in Multi-Modal Food Recipes

Modeling the structure of culinary recipes is the core of recipe representation learning. Current approaches mostly focus on extracting the workflow graph from recipes based on text descriptions. Process images, which constitute an important part of cooking recipes, has rarely been investigated in recipe structure modeling. We study this recipe structure problem from a multi-modal learning perspective, by proposing a prerequisite tree to represent recipes with cooking images at a step-level gran-ularity. We propose a simple-yet-effective two-stage framework to automatically construct the prerequisite tree for a recipe by (1) utilizing a trained classifier to detect pairwise prerequisite relations that fuses multi-modal features as input; then (2) applying different strategies (greedy method, maximum weight, and beam search) to build the tree structure. Experiments on the MM-ReS dataset demonstrates the advantages of introducing process images for recipe structure modeling. Also, compared with neural methods which require large numbers of training data, we show that our two-stage pipeline can achieve promising results using only 400 labeled prerequisite trees as training data.

Process Mining: Realization and Optimization of Process Discovery Algorithm

The article discusses the approach to the analysis of business processes, called process mining, and the directions of its application. In particular, a description of the process discovery algorithm is provided, with the help of which the process model is reconstructed from the event log in the form of a workflow graph. The implementation and improvement of the algorithm are proposed, which allow avoiding costly enumeration of all subsets of activities included in the analyzed process model. The proposed approach allows the use of process discovery algorithm for large event logs.

Formal Analysis of Segregation of Duties (SoD) in Accounting: A Computational Approach

This study examines a computational framework for segregation of duties (SoD) in the design as well as implementation of accounting systems. The framework consists of a model of workflows in accounting systems based on workflow graphs, a partial order model of roles performed by the actors in the accounting system, and a specification of SoD rules. We develop a set of algorithms for four SoD rules that can be used in the enforcement of SoD. For the SoD rule that precludes task type conflicts, our results show that while compliance verification can be carried out efficiently, finding an SoD compliant assignment of tasks is computationally intractable. For those situations, we present an integer linear programming (ILP) formulation for finding compliant assignments using public domain ILP solvers. For the remaining three SoD rules, we demonstrate efficient ways of testing compliance for a given assignment as well as finding compliant assignments.

A Data and Task Co-Scheduling Algorithm for Scientific Cloud Workflows

Cloud computing has emerged as a promising computational infrastructure for cost-efficient workflow execution by provisioning on-demand resources in a pay-as-you-go manner. While scientific workflows require accessing community-wide resources, they usually need to be performed in collaborative cloud environments composed of multiple datacenters. Although such environments facilitate scientific collaboration, the movements of input and intermediate datasets across geographically distributed datacenters may cause intolerable latency that would hinder efficient execution of large-scale data-intensive scientific workflows. To address the problem, in this article we propose a novel multi-level K-cut graph partitioning algorithm to minimize the volume of data transfer across datacenters while satisfying load balancing and fixed data constraints. The algorithm first contracts the fixed input datasets in the same datacenter and their consuming tasks, and coarsens the contracted graph to a predefined scale in a level-by-level manner. Then, a K-cut algorithm is used to partition the resulted graph into K parts such that the cut size is minimized. After that, the partitioned graph is projected back to the original workflow graph, during which the load balancing constraint is maintained. We evaluate our algorithm using three real-world workflow applications and the results demonstrate that the proposed algorithm outperforms other state-of-the-art algorithms.

Algorithmic node classification in AND/OR mobile workflow graph

The data-intensive applications employ complex workflow graph models in dynamic networks having heterogeneity and mobility of nodes, which result in enhanced end-to-end delay in executions. The current workflow models lack structural stability for visualising a complex workflow graph. This paper proposes a hybrid AND/OR mobile workflow graph (MWG) model to visualise a fully conditioned complex workflow graph containing mobile nodes. This paper proposes nodes validity detection (NVD) algorithm for classifying the total number of nodes in the AND/OR MWG. Furthermore, nodes criticality detection (NCD) algorithm is also proposed to identify the set of critical nodes in the AND/OR MWG aiming to enable efficient analysing, mapping and scheduling of complex workflow graphs in a dynamic network environment. The algorithms are implemented and evaluated on Java platform and, the regression analysis is performed to project the algorithmic performances. A detailed comparative analysis with other works is presented in this paper.

Algorithmic node classification in AND/OR mobile workflow graph

The data-intensive applications employ complex workflow graph models in dynamic networks having heterogeneity and mobility of nodes, which result in enhanced end-to-end delay in executions. The current workflow models lack structural stability for visualising a complex workflow graph. This paper proposes a hybrid AND/OR mobile workflow graph (MWG) model to visualise a fully conditioned complex workflow graph containing mobile nodes. This paper proposes nodes validity detection (NVD) algorithm for classifying the total number of nodes in the AND/OR MWG. Furthermore, nodes criticality detection (NCD) algorithm is also proposed to identify the set of critical nodes in the AND/OR MWG aiming to enable efficient analysing, mapping and scheduling of complex workflow graphs in a dynamic network environment. The algorithms are implemented and evaluated on Java platform and, the regression analysis is performed to project the algorithmic performances. A detailed comparative analysis with other works is presented in this paper.

EERA: An Energy-Efficient Resource Allocation Strategy for Mobile Cloud Workflows

Mobile cloud computing (MCC) which can invoke cloud services and offload tasks from the mobile device to the cloud has become an appropriate computing paradigm to provide many useful and complex workflow applications. However, offloading tasks needs extra communication time and energy, which leads to a conflict between saving the energy and improving the QoS. Thus, how to allocate these different resources (mobile or cloud) efficiently during the workflow execution process to optimize the system energy consumption under the deadline constraint is still a huge challenge in MCC. To address such an issue, this article proposes an energy-efficient resource allocation strategy for workflow applications in MCC. Firstly, we formulate the resource allocation problem into an optimization problem with the goal of minimizing the UtilityCost which represents the trade-off between energy consumption and the total execution time. Then, we use an energy-efficient resource allocation algorithm shortened as EERA to solve the problem. The algorithm firstly finds all the Partial critical paths of the workflow graph to segment the whole optimization problem into several local problems, and then uses the discrete particle swarm optimization algorithm to find the local optimal solution for every local problem, finally the local optimal solution is integrated to obtain the global one. In addition, we have adapted several representative algorithms for evaluation and comparison purpose. Comprehensive experimental results show that our EERA based strategy achieves the best overall performance on three key measurements including the energy consumption of the device, the execution time of the workflow and the UtilityCost.

Isolating critical flow path and algorithmic partitioning of the AND/OR mobile workflow graph

The standard workflow modeling approaches have insufficient information about structural stability to visualize the tasks in a complex workflow graph. The hybrid AND/OR mobile workflow graph (MWG) model addresses this issue by visualizing a fully conditioned workflow graph for dynamic networks. However, the scheduling and mapping of large scale workflows to heterogeneous resources is NP-complete in parallel and distributed computing systems. This paper aims to partition large scale workflows into mappable sub-workflows by isolating the critical flow paths using a filter matrix method. A set of algorithms are designed to perform decomposability checking and partitions of a AND/OR mobile workflow graph. The filter matrix allows input of user’s constraints prior to the partitioning of a complex workflow graph. The experimental results show that valid partitions are obtained from various AND/OR mobile workflow graph. Moreover, non-linear regression analysis is presented to project the algorithmic time complexity for the partitioning of the large scale workflow graphs. A detailed comparative analysis of the proposed algorithm is included in the paper.

Profiling the scheduling decisions for handling critical paths in deadline-constrained cloud workflows

Abstract In this paper, we study the scheduling decisions for handling deadline-constrained workflows in the context of planning customized virtual infrastructures in the cloud. We specifically focus on the effects of using different types of greediness in selecting cost-effective virtual machines for the tasks in an application’s workflow graph. The profiling procedure followed demonstrates that for the widely used approach of the partial critical path algorithm a greedy version is preferred to a more stringent version under different stress conditions, from tight to loose deadlines. Representative topologies of workflow applications are used to generate sets of task graph scheduling problems. Monitoring the performance of the partial critical path algorithm with different types of greediness reveals which of the topologies tested are difficult to solve under various stress conditions. It turns out that an invalid outcome of a greedy version of the partial critical path algorithm is more susceptible to become valid via a final refinement cycle than a less greedy version. The procedure outlined in this paper will allow for a systematic study of a specific heuristic in a workflow scheduling method to increase its success in infrastructure planning under different deadline conditions and is proposed to be part of a general profiling framework.

Reinforcement Learning based scheduling in a workflow management system

Any computational process from simple data analytics tasks to training a machine learning model can be described by a workflow. Many workflow management systems (WMS) exist that undertake the task of scheduling workflows across distributed computational resources. In this work, we introduce a WMS that leverages machine learning to predict workflow task runtime and the probability of failure of task assignments to execution sites. The expected runtime of workflow tasks can be used to approximate the weight of the workflow graph branches in respect to the total workflow workload and the ability to anticipate task failures can discourage task assignments that are unlikely to succeed. We demonstrate that the proposed machine learning models can lead to significantly more informed scheduling decisions that minimize task failures and utilize execution sites more efficiently, thus leading to reduced workflow runtime. Additionally, we train a modified sequence-to-sequence neural network architecture via reinforcement learning to perform scheduling decisions as part of a WMS. Our approach introduces a WMS that can drastically improve its scheduling performance by independently learning over time, without external intervention or reliance on any specific heuristic or optimization technique. Finally, we test our approach in real-world scenarios utilizing computationally demanding and data intensive workflows and evaluate its performance against existing scheduling methodologies traditionally used in WMSes. The performance evaluation outcome confirms that the proposed approach significantly outperforms the other scheduling algorithms in a consistent manner and achieves the best execution runtime with the lowest number of failed tasks and communication costs.

PL-DVFS: combining Power-aware List-based scheduling algorithm with DVFS technique for real-time tasks in Cloud Computing

In recent years, energy efficiency has emerged as one of the most important design requirements for modern computing systems, ranging from single servers to data centers and Clouds, as they continue to consume an enormous amount of electrical power. Cloud computing can be used to achieve energy efficiency through efficient task scheduling in the distributed environment. This efficient task scheduling helps to improve resource utilization, which, in turn, helps to minimize energy consumption. In this paper, we work toward minimizing energy of directed acyclic graph-structured applications on heterogeneous cloud system. The paper also combines power-aware list-based scheduling algorithm with dynamic voltage and frequency scaling (DVFS) technique for real-time tasks (PL-DVFS) to maintain the quality of service while considering tasks deadlines. The goal of the approach is to improve performance and overall reduced energy consumption comprising CPU energy (busy and idle) and communication energy. Experiments conducted with synthetic workflow graphs clearly demonstrate the advantage of the proposed approach.