Cloud Workflow Systems Research Articles

Runtime Verification of Business Cloud Workflow Temporal Conformance

Business cloud workflows are often designed with multiple time constraints for timely response to business requests. To ensure on-time completion of workflow instances, workflow temporal conformance state needs to be constantly monitored and verified at runtime. Considering the fact that there are a large number of workflow instances running in a parallel fashion in many business scenarios, conventional verification approaches for time-related properties using such as temporal logic or timed Petri nets are not feasible due to the limitation of low efficiency at runtime. To address this issue, we propose a new approach to automated runtime verification of temporal conformance for parallel workflow instances in a cloud environment. Instead of using response time to verify temporal conformance of every single workflow as in conventional strategies, workflow throughput is employed as the performance measurement to efficiently monitor a large number of parallel workflow instances. On this basis we present a novel conformance verification strategy. This strategy considers the effect of time delay propagation in the cloud workflow systems to accurately verify workflow runtime temporal conformance. Our verification strategy is implemented in a prototype cloud workflow system and the evaluation results show that it outperforms the state-of-the-art workflow temporal verification strategy.

Survey on Key Technologies in Cloud Workflow System Integration

Cloud computing has many unique advantages which can be utilized to facilitate workflow execution. The goal in cloud computing is to allocate only those cloud resources that are truly needed. In this paper, we introduced the cloud workflow system architecture based on Muti Agent(MAS),designed interface agent based on reflective factory pattern, defined paramters of quality of service (QoS) in cloud workflow system and gived QoS’s estimation mode of activitis. The research of the above key technologies Provides a reliable guarantee for the scheduling of the cloud workflow process.

Efficiently querying large process model repositories in smart city cloud workflow systems based on quantitative ordering relations

With the development of cloud computing and the rise of smart city, smart city cloud service platforms are widely accepted by more and more enterprises and individuals. The underlying cloud workflow systems accumulate large numbers of business process models. How to achieve efficiently querying large process model repositories in smart city cloud workflow systems is challenging. To this end, this paper proposes an improved two-phase retrieval approach for querying large process model repositories in smart city cloud workflow systems. In the filtering stage, the index based on quantitative ordering relation with time and probability constraints (namely ORTP_index) is adopted to greatly reduce the number of candidate models in large process model repositories. In the refining phase, a process behavior similarity computing algorithm based on quantitative ordering relations is proposed to refine the candidate model set. Experiments illustrate that our proposal can significantly improve the query efficiency of large process model repositories in smart city cloud workflow systems based on behavior.

A Novel Workflow-Level Data Placement Strategy for Data-Sharing Scientific Cloud Workflows

Cloud computing can provide a more cost-effective way to deploy scientific workflows than traditional distributed computing environments such as cluster and grid. Due to the large size of scientific datasets, data placement plays an important role in scientific cloud workflow systems for improving system performance and reducing data transfer cost. Traditional task-level data placement strategy only considers shared datasets within individual workflows to reduce data transfer cost. However, it is obvious that task-level strategy is not necessarily good enough for the situation of multiple workflows at the workflow level. In this paper, a novel workflow-level data placement model is constructed, which regards multiple workflows as a whole. Then, a two-stage data placement strategy is proposed which first pre-allocates initial datasets to proper datacenters during workflow build-time stage, and then dynamically distributes newly generated datasets to appropriate datacenters during runtime stage. Both stages use an efficient discrete particle swarm optimization algorithm to place flexible-location datasets. Comprehensive experiments demonstrate that our workflow-level data placement strategy can be more cost-effective than its task-level counterpart for data-sharing scientific cloud workflows.

Scientific workflow execution system based on mimic defense in the cloud environment

With more large-scale scientific computing tasks being delivered to cloud computing platforms, cloud workflow systems are designed for managing and arranging these complicated tasks. However, multi-tenant coexistence service mode of cloud computing brings serious security risks, which will threaten the normal execution of cloud workflows. To strengthen the security of cloud workflows, a mimic cloud computing task execution system for scientific workflows is proposed. The idea of mimic defense contains mainly three aspects: heterogeneity, redundancy, and dynamics. For heterogeneity, the diversities of physical servers, hypervisors, and operating systems are integrated to build a robust system framework. For redundancy, each sub-task of the workflow will be executed simultaneously by multiple executors. Considering efficiency and security, a delayed decision mechanism is proposed to check the results of task execution. For dynamics, a dynamic task scheduling mechanism is devised for switching workflow execution environment and shortening the life cycle of executors, which can confuse the adversaries and purify task executors. Experimental results show that the proposed system can effectively strengthen the security of cloud workflow execution.

Application of Workflow Technology for Big Data Analysis Service

This study presents a lightweight representational state transfer-based cloud workflow system to construct a big data intelligent software-as-a-service (SaaS) platform. The system supports the dynamic construction and operation of an intelligent data analysis application, and realizes rapid development and flexible deployment of the business analysis process that can improve the interaction and response time of the process. The proposed system integrates offline-batch and online-streaming analysis models that allow users to conduct batch and streaming computing simultaneously. Users can rend cloud capabilities and customize a set of big data analysis applications in the form of workflow processes. This study elucidates the architecture and application modeling, customization, dynamic construction, and scheduling of a cloud workflow system. A chain workflow foundation mechanism is proposed to combine several analysis components into a chain component that can promote efficiency. Four practical application cases are provided to verify the analysis capability of the system. Experimental results show that the proposed system can support multiple users in accessing the system concurrently and effectively uses data analysis algorithms. The proposed SaaS workflow system has been used in network operators and has achieved good results.

Predicting temporal violations for parallel business cloud workflows

SummaryWorkflow temporal violations, namely, intermediate workflow runtime delays, often occur and have a serious impact on the on‐time completion of massive concurrent requests. Therefore, accurate prediction of cloud workflow temporal violations is critical as its result can serve as an essential reference for temporal violation prevention and handling strategies. Conventional studies mainly focus on the time delays of a single workflow activity or a single workflow instance but overlook the propagation of time delays among them. This is a serious problem as time delays can propagate in cloud workflow system due to resource sharing and the dependencies among workflow activities. This paper first proposes a novel temporal violation transmission model inspired by an epidemic model to model the dynamics of time delay propagation. Afterward, a novel temporal violation prediction strategy is presented to estimate the number of temporal violations that may occur and determine the number of violations that must be handled to achieve the target service‐level agreement, namely, the on‐time completion rate. To the best of our knowledge, this is the first attempt to predict cloud workflow temporal violations at the workflow build‐time stage by analyzing the propagation of temporal violations. Experimental results demonstrate that our strategy can make highly accurate predictions and is scalable for a large batch of parallel workflows running in the cloud.

SCW+: A Service Oriented Framework for Cloud Workflow Systems

Cloud workflow is a special type of cloud computing systems which mainly concentrates on workflow management. One of the major issues with cloud workflow systems is automatic multi-cloud workflow management. This paper proposes a service oriented framework for cloud workflow management which integrates heterogeneous multi-cloud platforms to provide integrated applications for users by minimizing the human intervention as far as possible. The proposed framework involves a language to define some basic entities for environments and uses these definitions to integrate applications and services in a cloud workflow. This framework has been already implemented. In addition, its main operations have been evaluated by a case study and the results show that the framework works properly as a cloud integrator, and the framework main activities are done automatically with a reasonable performance.

A Dynamic Pricing Reverse Auction-Based Resource Allocation Mechanism in Cloud Workflow Systems

Market-oriented reverse auction is an efficient and cost-effective method for resource allocation in cloud workflow systems since it can dynamically allocate resources depending on the supply-demand relationship of the cloud market. However, during the auction the price of cloud resource is usually fixed, and the current resource allocation mechanisms cannot adapt to the changeable market properly which results in the low efficiency of resource utilization. To address such a problem, a dynamic pricing reverse auction-based resource allocation mechanism is proposed. During the auction, resource providers can change prices according to the trading situation so that our novel mechanism can increase the chances of making a deal and improve efficiency of resource utilization. In addition, resource providers can improve their competitiveness in the market by lowering prices, and thus users can obtain cheaper resources in shorter time which would decrease monetary cost and completion time for workflow execution. Experiments with different situations and problem sizes are conducted for dynamic pricing-based allocation mechanism (DPAM) on resource utilization and the measurement of Time⁎Cost (TC). The results show that our DPAM can outperform its representative in resource utilization, monetary cost, and completion time and also obtain the optimal price reduction rates.

Necessary and sufficient checkpoint selection for temporal verification of high-confidence cloud workflow systems

On-time completion is an important temporal QoS (Quality of Service) dimension and one of the fundamental requirements for high-confidence workflow systems. In recent years, a workflow temporal verification framework, which generally consists of temporal constraint setting, temporal checkpoint selection, temporal verification, and temporal violation handling, has been the major approach for the high temporal QoS assurance of workflow systems. Among them, effective temporal checkpoint selection, which aims to timely detect intermediate temporal violations along workflow execution plays a critical role. Therefore, temporal checkpoint selection has been a major topic and has attracted significant efforts. In this paper, we will present an overview of work-flow temporal checkpoint selection for temporal verification. Specifically, we will first introduce the throughput based and response-time based temporal consistency models for business and scientific cloud workflow systems, respectively. Then the corresponding benchmarking checkpoint selection strategies that satisfy the property of“necessity and sufficiency”are presented. We also provide experimental results to demonstrate the effectiveness of our checkpoint selection strategies, and finally points out some possible future issues in this research area.

A Chaotic Particle Swarm Optimization-Based Heuristic for Market-Oriented Task-Level Scheduling in Cloud Workflow Systems.

Cloud workflow system is a kind of platform service based on cloud computing. It facilitates the automation of workflow applications. Between cloud workflow system and its counterparts, market-oriented business model is one of the most prominent factors. The optimization of task-level scheduling in cloud workflow system is a hot topic. As the scheduling is a NP problem, Ant Colony Optimization (ACO) and Particle Swarm Optimization (PSO) have been proposed to optimize the cost. However, they have the characteristic of premature convergence in optimization process and therefore cannot effectively reduce the cost. To solve these problems, Chaotic Particle Swarm Optimization (CPSO) algorithm with chaotic sequence and adaptive inertia weight factor is applied to present the task-level scheduling. Chaotic sequence with high randomness improves the diversity of solutions, and its regularity assures a good global convergence. Adaptive inertia weight factor depends on the estimate value of cost. It makes the scheduling avoid premature convergence by properly balancing between global and local exploration. The experimental simulation shows that the cost obtained by our scheduling is always lower than the other two representative counterparts.

Efficient and exact query of large process model repositories in cloud workflow systems

As cloud computing platforms are widely accepted by more and more enterprises and individuals, the underlying cloud workflow systems accumulate large numbers of business process models. Retrieving and recommending the most similar process models according to the tenant's requirements become extremely important, for it is not only beneficial to promote the reuse of the existing model assets, but also helpful to reduce the error rate of the modeling process. Since the scales of cloud workflow repositories become bigger and bigger, developing efficient and exact query approaches is urgent. To this end, an improved two-stage exact query approach based on graph structure is proposed. In the filtering stage, the composite task index , which consists of the label, join-attribute and split-attribute of a task, is adopted to acquire candidate models, which can greatly reduce the number of process models needed to be tested by a time-consuming verification algorithm. In the verification stage, a novel subgraph isomorphism test based on task code is proposed to refine the candidate model set. Experiments are conducted on six synthetic model sets and two real model sets. The results demonstrate that the presented approach can significantly improve the query efficiency and reduce the query response time.

A dataflow optimisation mechanism for service-oriented cloud workflow

While service-oriented cloud workflow shows potentials of inherent scalability, loose coupling and expenditure reduction, such issues as data transfer and efficiency have popped up as major concerns. In this paper, a dataflow optimisation mechanism is proposed, including data transfer strategy and data placement strategy. The data transfer strategy uses integrated data structures to enable the elimination of the SOAP serialisation and de-serialisation within the web service invocation by the exchange of their references. In addition, the data placement strategy groups the existing datasets and dynamically clusters newly generated datasets to minimise data transfers while keeping a polynomial time complexity. By integrating the optimisation mechanism into service-oriented cloud workflow system, we can expect efficiency improvements in data transfer and workflow execution. Experiments and analysis supported our efforts.

Queuing Mechanism in Migrating Workflow System Based on Cloud Computing Paradigm (MWfSCC)

Migrating workflow system based on cloud computing paradigm can be regarded as a type of platform service which facilitates the automation of distributed large-scale e-business and e-science applications in the cloud computing platform. CMI is a software entity that can migrate from one site to another in order to satisfy its goals in cloud workflow system. However, if a large number of CMIs migrate to one work machine within a short period of time, congestion is so serious that the average executive time for CMIs would be much longer due to the competition for the limited resources. In this paper, we introduce the queuing conception into cloud workflow system to solve this problem. Two queuing models are created for corresponding two different types of work machine, called sole work-node model and compound work-node model. For each model, a set of detailed queuing rules, service rules and the CMI dispatching mechanism are defined. We choose P/N/m/∞ queuing model to implement queuing mechanism in sole work-node model, and use dynamic priority queue to manage waiting CMI. Compound work-node is made up of a lot of sole work-nodes, when a CMI migrated to one compound work-node, it would be dispatched to one sole-work node according to the task type to be executed. The effectiveness of the queuing mechanism is evident in that it can alleviate the load of work nodes, which are illustrated with the experiment results at the end of paper.

Service Selection Algorithm Based on Constraint for Cloud Workflow System

It is possible to utilize cloud computing to address the problems of resource extendibility and flexibility for managing large scale workflow applications. Therefore, the investigation of workflow systems based on cloud computing, namely cloud workflow systems is a timely issue and worthwhile for increasing efforts. The technology and application of workflow in cloud computing environment also becomes a research focus. The paper firstly proposes a brief introduction to the cloud workflow system. Next, based on cloud architecture, the paper presents the architecture of cloud workflow system. Then, as the key content of the paper, considering the service selection strategy based on constraint, the paper presents a new service selection algorithm–SSAC. The simulation results show that SSAC algorithm further decreases the total execution cost and total execution time in comparison with other algorithms.

Design and Optimization of Cloud-Oriented Workflow System

The paper firstly analyzes the characteristic of existing workflow applications and proposes cloud workflow layered model and design method based on the current major cloud computing trends. Considering the features on the cloud workflow system in distributed heterogeneous environment, the paper proposes a brief introduction to approach to cloud workflow system. Next, three deployment models of the cloud workflow applications are presented. Finally, the paper presents a new dynamic scheduling algorithm –Improved Dynamic Cloud Scheduling Algorithm(IDCSA) which takes consideration of the characteristics of cloud workflow and cloud resources.

A market-oriented hierarchical scheduling strategy in cloud workflow systems

A cloud workflow system is a type of platform service which facilitates the automation of distributed applications based on the novel cloud infrastructure. One of the most important aspects which differentiate a cloud workflow system from its other counterparts is the market-oriented business model. This is a significant innovation which brings many challenges to conventional workflow scheduling strategies. To investigate such an issue, this paper proposes a market-oriented hierarchical scheduling strategy in cloud workflow systems. Specifically, the service-level scheduling deals with the Task-to-Service assignment where tasks of individual workflow instances are mapped to cloud services in the global cloud markets based on their functional and non-functional QoS requirements; the task-level scheduling deals with the optimisation of the Task-to-VM (virtual machine) assignment in local cloud data centres where the overall running cost of cloud workflow systems will be minimised given the satisfaction of QoS constraints for individual tasks. Based on our hierarchical scheduling strategy, a package based random scheduling algorithm is presented as the candidate service-level scheduling algorithm and three representative metaheuristic based scheduling algorithms including genetic algorithm (GA), ant colony optimisation (ACO), and particle swarm optimisation (PSO) are adapted, implemented and analysed as the candidate task-level scheduling algorithms. The hierarchical scheduling strategy is being implemented in our SwinDeW-C cloud workflow system and demonstrating satisfactory performance. Meanwhile, the experimental results show that the overall performance of ACO based scheduling algorithm is better than others on three basic measurements: the optimisation rate on makespan, the optimisation rate on cost and the CPU time.

On-demand minimum cost benchmarking for intermediate dataset storage in scientific cloud workflow systems

Many scientific workflows are data intensive: large volumes of intermediate datasets are generated during their execution. Some valuable intermediate datasets need to be stored for sharing or reuse. Traditionally, they are selectively stored according to the system storage capacity, determined manually. As doing science on clouds has become popular nowadays, more intermediate datasets in scientific cloud workflows can be stored by different storage strategies based on a pay-as-you-go model. In this paper, we build an intermediate data dependency graph (IDG) from the data provenances in scientific workflows. With the IDG, deleted intermediate datasets can be regenerated, and as such we develop a novel algorithm that can find a minimum cost storage strategy for the intermediate datasets in scientific cloud workflow systems. The strategy achieves the best trade-off of computation cost and storage cost by automatically storing the most appropriate intermediate datasets in the cloud storage. This strategy can be utilised on demand as a minimum cost benchmark for all other intermediate dataset storage strategies in the cloud. We utilise Amazon clouds’ cost model and apply the algorithm to general random as well as specific astrophysics pulsar searching scientific workflows for evaluation. The results show that benchmarking effectively demonstrates the cost effectiveness over other representative storage strategies.

A data dependency based strategy for intermediate data storage in scientific cloud workflow systems

SUMMARYMany scientific workflows are data intensive where large volumes of intermediate data are generated during their execution. Some valuable intermediate data need to be stored for sharing or reuse. Traditionally, they are selectively stored according to the system storage capacity, determined manually. As doing science in the cloud has become popular nowadays, more intermediate data can be stored in scientific cloud workflows based on a pay‐for‐use model. In this paper, we build an intermediate data dependency graph (IDG) from the data provenance in scientific workflows. With the IDG, deleted intermediate data can be regenerated, and as such we develop a novel intermediate data storage strategy that can reduce the cost of scientific cloud workflow systems by automatically storing appropriate intermediate data sets with one cloud service provider. The strategy has significant research merits, i.e. it achieves a cost‐effective trade‐off of computation cost and storage cost and is not strongly impacted by the forecasting inaccuracy of data sets' usages. Meanwhile, the strategy also takes the users' tolerance of data accessing delay into consideration. We utilize Amazon's cost model and apply the strategy to general random as well as specific astrophysics pulsar searching scientific workflows for evaluation. The results show that our strategy can reduce the overall cost of scientific cloud workflow execution significantly. Copyright © 2010 John Wiley & Sons, Ltd.