Distributed Workflow Management System Research Articles

A multi‐tenancy and robust workflow management system

AbstractWorkflow management system (WfMS) in cloud always works as platform as a service to manage customized business processes for massive enterprises. In big data era, non‐functional guarantees of such systems are significant when facing a large number of users and concurrent requests. It is not trivial to support multi‐tenancy and hold high‐availability, because traditional architecture cannot simultaneously satisfy requirements about data isolation and runtime efficiency. In this paper, a modularized distributed workflow management system is proposed, which considers both multi‐tenancy and high‐availability in storage and engine parts of the system. A multiple‐worker‐with‐separate‐schema mechanism is defined to jointly manage the data for tenants, and a proactive strategy is presented to intelligently dispatch large concurrent requests from users to engine workers. After extensive case studies and experiments in practical scenes, our system deployed on modest machines is proved to support tens of thousands of tenants, second‐level response time for 10 K concurrency, and no‐human‐intervened failure recovery for a fail‐stop system node.

Tool Support for Distributed Workflow Management with Task Clustering

When in need for executing complex sets of interrelated calculations on High-Performance Computing (HPC) environments the obvious choice is to use scientific workflows. As workload management software do not support the execution of interrelated tasks, workflow management systems have been introduced to execute workflows on HPC environments. Recently, a new distributed architectural model that offers dynamic workflow execution capabilities to workflow management systems is introduced. It executes workflows on a per-task basis. While this approach facilitates dynamic workflows, it adds a considerable overhead to workflows substantially increasing their makespans. As most workflows are static, task-wise execution of workflows degrades the performance of most workflows. In this paper, we introduce a distributed workflow management system, SwarmForm that introduces task clustering to the new architectural model. SwarmForm is open source and offers better performance than existing distributed workflow management systems by clustering workflow tasks to reduce overheads while allowing the users to choose between task-wise and cluster-wise execution of workflows depending on the workflow nature. The paper proves that SwarmForm enables the use of all the features introduced with the new architectural model while providing better makespans for scientific workflows.

Research and Implementation of Key Technologies in Multi-Agent System to Support Distributed Workflow

In recent years, the combination of workflow management system and Multi-agent technology is a hot research field. The problem of lack of flexibility in workflow management system can be improved by introducing multi-agent collaborative management. The workflow management system adopts distributed structure. It solves the problem that the traditional centralized workflow structure is fragile. In this paper, the agent of Distributed workflow management system is divided according to its function. The execution process of each type of agent is analyzed. The key technologies such as process execution and resource management are analyzed.

Possibilistic Information Flow Control for Workflow Management Systems

In workflows and business processes, there are often security requirements on both the data, i.e. confidentiality and integrity, and the process, e.g. separation of duty. Graphical notations exist for specifying both workflows and associated security requirements. We present an approach for formally verifying that a workflow satisfies such security requirements. For this purpose, we define the semantics of a workflow as a state-event system and formalise security properties in a trace-based way, i.e. on an abstract level without depending on details of enforcement mechanisms such as Role-Based Access Control (RBAC). This formal model then allows us to build upon well-known verification techniques for information flow control. We describe how a compositional verification methodology for possibilistic information flow can be adapted to verify that a specification of a distributed workflow management system satisfies security requirements on both data and processes.

Workflow Systems for Science: Concepts and Tools

The wide availability of high-performance computing systems, Grids and Clouds, allowed scientists and engineers to implement more and more complex applications to access and process large data repositories and run scientific experiments in silico on distributed computing platforms. Most of these applications are designed as workflows that include data analysis, scientific computation methods, and complex simulation techniques. Scientific applications require tools and high-level mechanisms for designing and executing complex workflows. For this reason, in the past years, many efforts have been devoted towards the development of distributed workflow management systems for scientific applications. This paper discusses basic concepts of scientific workflows and presents workflow system tools and frameworks used today for the implementation of application in science and engineering on high-performance computers and distributed systems. In particular, the paper reports on a selection of workflow systems largely used for solving scientific problems and discusses some open issues and research challenges in the area.

A Distributed Workflow Management System with Case Study of Real-life Scientific Applications on Grids

Next-generation scientific applications feature complex workflows comprised of many computing modules with intricate inter-module dependencies. Supporting such scientific workflows in wide-area networks especially Grids and optimizing their performance are crucial to the success of collaborative scientific discovery. We develop a Scientific Workflow Automation and Management Platform (SWAMP), which enables scientists to conveniently assemble, execute, monitor, control, and steer computing workflows in distributed environments via a unified web-based user interface. The SWAMP architecture is built entirely on a seamless composition of web services: the functionalities of its own are provided and its interactions with other tools or systems are enabled through web services for easy access over standard Internet protocols while being independent of different platforms and programming languages. SWAMP also incorporates a class of efficient workflow mapping schemes to achieve optimal end-to-end performance based on rigorous performance modeling and algorithm design. The performance superiority of SWAMP over existing workflow mapping schemes is justified by extensive simulations, and the system efficacy is illustrated by large-scale experiments on real-life scientific workflows for climate modeling through effective system implementation, deployment, and testing on the Open Science Grid.

Flexible Distributed Workflow Management Systems Design Based on CORBA

The paper addresses the problem of flexible Workflow Management Systems (WFMS) in distributed environment. Concerning the serious deficiency of flexibility in the current workflow systems, we describe how our workflow system meets the requirements of interoperability, scalability, flexibility, dependability and adaptability. With an additional route engine, the execution path will be adjusted dynamically according to the execution conditions so as to improve the flexibility and dependability of the system. A dynamic register mechanism of domain engines is introduced to improve the scalability and adaptability of the system. The system is general purpose and open: it has been designed and implemented as a set of CORBA services. The system serves as an example of the use of middleware technologies to provide a fault-tolerant execution environment for long running distributed applications. The system also provides a mechanism for communication of distributed components in order to support inter-organizational WFMS.

On Secure Workflow Decentralisation on the Internet

Decentralised workflow management systems are a new research area, where most work to-date has focused on the system's overall architecture. As little attention has been given to the security aspects in such systems, we follow a security driven approach, and consider, from the perspective of available security building blocks, how security can be implemented and what new opportunities are presented when empowering the decentralised environment with modern distributed security protocols. Our research is motivated by a more general question of how to combine the positive enablers that email exchange enjoys, with the general benefits of workflow systems, and more specifically with the benefits that can be introduced in a decentralised environment. This aims to equip email users with a set of tools to manage the semantics of a message exchange, contents, participants and their roles in the exchange in an environment that provides inherent assurances of security and privacy. This work is based on a survey of contemporary distributed security protocols, and considers how these protocols could be used in implementing a distributed workflow management system with decentralised control . We review a set of these protocols, focusing on the required message sequences in reviewing the protocols, and discuss how these security protocols provide the foundations for implementing core control-flow, data, and resource patterns in a distributed workflow environment.

An integrated time management model for distributed workflow management systems in Grid environments

AbstractMulti‐granularity of time and time zone difference are two aspects of a time management model (TMM) for distributed workflow management system (DWfS). There are some recent literatures concerning each of them. However, current researches on DWfS have not put them together but have built up an integrated model. As business and scientific collaboration processes across continents become more and more frequent, it is increasingly important to build an integrated model to take them as a whole to make the time transformation and temporal verification in collaboration processes easy and efficient. Aiming at solving this problem, an integrated TMM, DWfS‐TMM, is proposed in this paper. The DWfS‐TMM model consists of a set of general time ontology and a set of general rules for unified transformation between different time zones and different time granularity units. The representation of build‐time and run‐time temporal constraints in workflow processes and the temporal consistency checking method in this model is investigated. Finally, a real case study is investigated and it is shown from this case study that the presented model is useful and practicable. Copyright © 2009 John Wiley & Sons, Ltd.

Interoperability of Cross-organizational Workflows based on Process-view for Collaborative Product Development

Collaborative product development (CPD) has been widely accepted as an advanced collaboration paradigm that combines geographically distributed product development teams to develop product collaboratively and efficiently. Interoperability of cross-organizational workflows is important for CPD to facilitate the successful execution of the whole product development process across enterprises boundaries. This article, describes a timed colored petri net (TCPN) and process-view combined approach to construct cross-organizational workflows, and a three models framework of `TCPN workflow models — Process-view workflow models — Integrated process-view workflow model' is proposed to realize the interoperability of cross-organizational workflows for CPD. A multi-agent system PVMAS provides a mediated architecture for distributed workflow management systems interoperability among cooperative enterprises. Under this architecture, a service discovery algorithm is provided for discovering the most suitable services enterprises need, a TCPN algebra model and a character string mapping algorithm are proposed to automate the mapping from a TCPN workflow model to a process-view workflow model, and the synchronization points are designated in the integrated process-view workflow model to coordinate the execution of cross-organizational workflow instances. Finally, a case study of the collaborative development of a motorcycle is presented to verify the validity of our approach for cross-organizational workflows interoperability in CPD.

Bridging Security and Fault Management within Distributed Workflow Management Systems

As opposed to centralized workflow management systems, the distributed execution of workflows can not rely on a trusted centralized point of coordination. As a result, basic security features including compliance of the overall sequence of workflow operations with the pre-defined workflow execution plan or traceability become critical issues that are yet to be addressed. Besides, the detection of security inconsistencies during the execution of a workflow usually implies the complete failure of the workflow although it may be possible in some situations to recover from the latter. In this paper, we present security solutions supporting the secure execution of distributed workflows. These mechanisms capitalize on onion encryption techniques and security policy models to assure the integrity of the distributed execution of workflows, to prevent business partners from being involved in a workflow instance forged by a malicious peer and to provide business partners identity traceability for sensitive workflow instances. Moreover, we specify how these security mechanisms can be combined with a transactional coordination framework to recover from faults that may be caught during their execution. The defined solutions can easily be integrated into distributed workflow management systems as our design is strongly coupled with the runtime specification of decentralized workflows.

Distributed Workflow Management System based on Publish-Subscribe Notification for Web Services

In recent years, Grid technologies have been standardized based on Web service specifications. Of these specifications, the WS-Resources Framework and WS-Notification have attracted a great deal of attention. This paper focuses on scientific applications integration on the wide area network. We propose and implement a new distributed workflow management system called the “Application Igniting System.” This system is based on the publish-subscribe notification defined by the WS-Notification specification and realizes a flexible and loosely coupled workflow control by introducing some Web services, which handle message exchange. By applying to a typical bioinformatics workflow, we concluded that the overhead time related to message exchange is very short.

Intra-Subnet Load Balancing in Distributed Workflow Management Systems

For enterprise-wide and cross-organizational process-oriented applications, the execution of workflows (WF) may generate a very high load. This load may affect WF servers as well as the underlying communication network. To improve system scalability, several approaches for distributed WF management have been proposed in the literature. They have in common that different partitions of a WF instance graph may be controlled by different WF servers from different subnets. The control over a particular WF instance, therefore, may be transferred from one WF server to another during run-time if this helps to reduce the overall communication load. Thus far, such distributed approaches assume that exactly one WF server resides in each subnet. A single server per subnet, however, may become overloaded. In this paper, we present and verify a novel approach for replicating WF servers in a distributed workflow management system. It enables an arbitrary and changeable distribution of the load to the WF servers of the same subnet, without requiring additional communication.

Autonomy versus efficiency in multi-agent management of extended enterprises

Present evolution of manufacturing systems is characterized by multi-firms enterprises, often organized in multi-site networks where each firm can be managed with large autonomy. Current workflow technology does not provide the necessary functionality to model and manage such processes, mostly due to centralized structures and limited process management capabilities. In this paper some on-going efforts to approach new problems involved in distributed decoupled management architectures, based on individual autonomous agents, are presented. In particular, the scope is to verify if and when the increasing complexity of a large-scale enterprise, composed by several firms which aim to cooperate together, is going to allow a greater and greater autonomy to local decision-makers. Based on new definitions of cooperation among the agents, the opportunity of employing decoupled management architecture is analyzed, thus also giving suggestions for distributed workflow management systems.

The distributed workflow management system — Flow agent

While mainframe or 2-tier client/server system have serious problems in flexibility and scalability for the large-scale business processes, 3-tier client/ server architecture and object-oriented system modeling which construct business process on service components seem to bring software system some scalability. As enabling infrastructure for object-oriented methodology, distributed WFMS (Work-flow Management System) can flexibly describe business rules among autonomous ‘service tasks’, and support scalability of large-scale business process. But current distributed WFMS still have difficulty to manage a large number of distributed tasks, the ‘multi-Task Domain’ architecture of FlowAgent will try to solve this problem, and bring a dynamic and distributed environment for task-scheduling.

A goal-driven auto-configuration tool for the distributed workflow management system mentorlite

The Mentor-lite prototype has been developed within the research project “Architecture, Configuration, and Administration of Large Workflow Management Systems” funded by the German Science Foundation (DFG). It has evolved from its predecessor Mentor [1], but aims at a simpler architecture. The main goal of Mentor-lite has been to build a light-weight, extensible, and tailorable workflow management system (WFMS) with small footprint and easy-to-use administration capabilities. Our approach is to provide only kernel functionality inside the workflow engine, and consider system components like history management and worklist management as extensions on top of the kernel. The key point to retain the light-weight nature is that these extensions are implemented as workflows themselves. The workflow specifications are interpreted at runtime, which is a crucial prerequisite for flexible exception handling and dynamic modifications during runtime. The interpreter performs a stepwise execution of the workflow specification according to its formal semantics. For each step, the activities to be performed by the step are determined and started. Mentor-lite supports a protocol for distributed execution of workflows spread across multiple workflow engines. This support is crucial for workflows that span large, decentralized enterprises with largely autonomous organizational units or even cross multiple enterprises to form so-called “virtual enterprises”. A communication manager is responsible for sending and receiving synchronization messages between the engines. In order to guarantee a consistent global state even in the presence of site or network failures, we have built reliable message queues using the CORBA Object Transaction Services. For administration, Mentor-lite provides a Java-based workbench for workflow design, workflow partitioning across multiple workflow servers, and a Java-based runtime monitoring tool.