Workflow Tasks Research Articles

Balanced scheduling of distributed workflow tasks based on clustering

Distributed computing, such as Cloud, provides traditional workflow applications with completely new deployment architecture offering high performance and scalability. However, when executing the workflow tasks in a distributed computing environment, significant scheduling overheads are generated. Task clustering is a key technology to optimize process execution. Unreasonable task clustering can lead to the problems of runtime and dependency imbalance, which reduces the degree of parallelism during task execution. In order to solve the problem of runtime imbalance, we propose Runtime Balance Clustering Algorithm (RBCA), which employs the Backtracking approach to make the runtime of each cluster more balanced. In addition, to address the problem of dependency imbalance, we also propose Dependency Balance Clustering Algorithm (DBCA), which defines the dependency correlation to measure the similarity between tasks in terms of data dependencies. The tasks with high dependency correlation are clustered together so as to avoid the dependency imbalance to most extent. We conducted extensive experiments on the WorkflowSim platform and compared our algorithms with the existing task clustering algorithms. The results show that RBCA and DBCA significantly reduce the execution time of the whole workflow.

A dynamic VM provisioning and de-provisioning based cost-efficient deadline-aware scheduling algorithm for Big Data workflow applications in a cloud environment

Cloud computing is the fastest growing distributed computing paradigm that provides online IT resources on demand by following a pay-as-you-go billing model. The success of this computing paradigm enables cloud providers to offer an extensive collection of parallel computing resources to deal with Big Data workflow scheduling problems. Although, workflow scheduling has been extensively studied, however, most of them are unable to achieve user-specified deadline constraints at the cheap cost. In this paper, a Dynamic Cost-Efficient Deadline-Aware (DCEDA) heuristic algorithm is proposed for scheduling Big Data workflow that produces the cheapest schedule while achieving the deadline constraints. DCEDA dynamically takes appropriate scheduling decisions for workflow tasks based on the fact that deadline constraint is not violated in the future. Also, it continuously monitors the VM pool for identifying the active idle VMs that incur extra costs and overheads, and subsequently de-provision them. The experimental analysis based on Montage workflow and randomly generated synthetic workflow with various characteristics prove that DCEDA delivers better performance in comparison to the existing algorithms.

A Survey on QoS Requirements Based on Particle Swarm Optimization Scheduling Techniques for Workflow Scheduling in Cloud Computing

Cloud computing is an innovative technology that deploys networks of servers, located in wide remote areas, for performing operations on a large amount of data. In cloud computing, a workflow model is used to represent different scientific and web applications. One of the main issues in this context is scheduling large workflows of tasks with scientific standards on the heterogeneous cloud environment. Other issues are particular to public cloud computing. These include the need for the user to be satisfied with the quality of service (QoS) parameters, such as scalability and reliability, as well as maximize the end-users resource utilization rate. This paper surveys scheduling algorithms based on particle swarm optimization (PSO). This is aimed at assisting users to decide on the most suitable QoS consideration for large workflows in infrastructure as a service (IaaS) cloud applications and mapping tasks to resources. Besides, the scheduling schemes are categorized according to the variant of the PSO algorithm implemented. Their objectives, characteristics, limitations and testing tools have also been highlighted. Finally, further directions for future research are identified.

Budget aware scheduling algorithm for workflow applications in IaaS clouds

Cloud computing, a novel and promising model of Service-oriented computing, provides a pay-per-use framework to solve large-scale scientific and business workflow applications. Workflow scheduling in cloud is challenging due to dynamic nature of the cloud, particularly, on demand provisioning, elasticity, heterogeneous resource types, static & dynamic pricing models and virtualization. An example of workflow scheduling is mapping workflow tasks to cloud computing resources. Additionally, these workflow applications have a runtime constraint—the most typical being the cost of the computation and the time that computation requires to complete. Therefore, the focus is on two criteria: makespan and cost. This paper presents an algorithm called NBWS (Normalization based Budget constraint Workflow Scheduling) which generates a workflow schedule which minimizes the schedule length while satisfying the budget constraint. The algorithm undergoes a process of min–max normalization tailed by computing expect reasonable budget $$ (erb) $$ for dispatching the workflow tasks into one of the virtual machines. To minimize the execution time, NBWS algorithm maps the workflow tasks to resources which are having the earliest finish time within the allocated budget. The experimental results demonstrate that NBWS outperforms current state-of-the-art heuristics with respect to budget constraint and minimizing the makespan.

Single-channel and multi-channel seismic random noise suppression based on the regularized non-stationary decomposition

Strong random noise existing in the seismic data often deteriorates many processing and interpretation tasks in the whole seismic imaging workflow. To tackle the challenge in processing noise-contaminated seismic data, we propose to apply a regularized non-stationary decomposition (RNSD) method to attenuate the strong noise in the seismic data. The RNSD method can not only be applied to denoise single-channel seismic data, e.g., micro-seismic data, but also be used to deal with multi-channel seismic data. The RNSD is formulated as an inverse problem, in which the noise is assumed to be the misfit within the inverse problem. When the RNSD method is applied to denoise single-channel seismic data, only temporal constraint is applied to invert the decomposing results. When the RNSD method is applied to denoise multi-channel seismic data, both time and space constraints are enforced. For the convenience of applying the constraints, we apply the shaping method to enforce the decomposed results to be either temporally smooth or bi-laterally smooth. We apply the proposed method to both synthetic and field data examples to illustrate the denoising power of the proposed method.

TOPSIS inspired cost-efficient concurrent workflow scheduling algorithm in cloud

Scheduling is a decision-making mechanism that enables the sharing of resources among several activities by determining their execution order on the set of available resources. In distributed systems, it is a great challenge to schedule multiple workflows submitted at different times. In particular, concurrent workflow scheduling with time constraints makes the problem more complex in the cloud due to the dynamics of the cloud such as elasticity, non-homogeneous resource types, various pricing schemes, and virtualization. A well-managed deadline workflow scheduling is required to improve end-user satisfaction and system performance. In the meantime, the intrinsic uncertainty in the cloud increases the difficulties of scheduling problems. Therefore, it is a great challenge to improve system performance and optimize several scheduling criteria simultaneously. To address the above issues, a novel concurrent workflow scheduling method for heterogeneous distributed environments based on the new Multi-Criteria Decision Making (MCDM) method i.e., TOPSIS (Technique of Order Preference by Similarity to Ideal Solution) is presented. A weighted sum of execution time, cost and communication time are used to find out the optimal resource among the existing resources as per the workflow task requirements. The proposed method minimizes the makespan and execution cost of the workflow and improves the resource efficiency under uncertain environment. The performance of the proposed work is compared with the state-of-the-art algorithms such as Cloud-based Workflow Scheduling Algorithm (CWSA), Earliest Finish Time-Maximum Effective Reduction (EFT-MER) and Heterogeneous Earliest-Finish-Time (HEFT) algorithms based on deadline constraint and resource utilization. Our experimental results demonstrate that the proposed T-CCWSA outperforms current state-of-the-art heuristics with the criteria of achieving the deadline constraint, minimizing the cost of execution and resource efficiency.

Self-adapting cloud services orchestration for fulfilling intensive sensory data-driven IoT workflows

Cloud computing has been adopted to support among others the storage and processing of complex Internet of Things (IoT) workflows handling sensory streamed time-series data. IoT workflow is often composed following a set of procedures which makes it hard to self-adapt, self-configure to react to runtime environment changes. Therefore, declarative data-driven workflow composition will provision self-learning and self-configurable workflows such as those of IoT. This paper proposes a comprehensive architecture to support end-to-end workflow management processes including declarative specification and composition, configuration deployment, orchestration, execution, adaptation, and quality enforcement. The later provision runtime intelligence for IoT workflow orchestration; this is achieved through the automated monitoring and analysis of runtime cloud resource orchestration, the monitoring of workflows tasks execution, as well as through cloud resource utilization prediction and workflow adaptation. In addition, it supports other intelligent features that include: (1) integration of edge computing (sensor edge) for local data processing which is very crucial for life-critical IoT workflows, (2) data compression for fast data transmission, and data storage adaptation, and (3) customization of data reporting and visualization. All these features have been evaluated through a set of experiments that proved a significant gain in terms of workflow execution time, cost and optimum usage of cloud resources compared to baseline adaptation strategy.

Big Data Processing Workflows Oriented Real-Time Scheduling Algorithm using Task-Duplication in Geo-Distributed Clouds

Scheduling big data processing workflows involves both large-scale tasks and transmission of massive intermediate data among tasks, thus optimizing their completion time and monetary cost becomes a challenging issue. Besides, data streams are continuously generated, and dynamically submitted to clouds for real-time or near real-time processing. Naturally, responsive schedules are required to keep pace with such dynamic environments and this further aggravates the difficulty of the workflow scheduling problem. To address these issues, we first derive two theorems to minimize the completion time of a set of parallel workflow tasks and the start time of each workflow task, and then define the latest finish time for workflow tasks, which is also proved its advantage in reducing costs without delaying the completion of workflows. On the basis of these theorems, we propose a novel r eal- t ime s cheduling a lgorithm using t ask- d uplication, RTSATD, such that minimizing both the completion time and monetary cost of processing big data workflows in clouds. The performance of RTSATD is analyzed by using both synthesized and real-world workflows. The experimental results demonstrate the superiority of the proposed algorithm with respect to completion time (up to 28.73 percent) and resource utilization (up to 46.31 percent) over two existing approaches.

Scheduling Workflows With Composite Tasks: A Nested Particle Swarm Optimization Approach

Scientific cloud workflows enable the access to distributed computing resources in cloud environments for executing scientific computing applications. In the literature, most workflow scheduling models assume that each workflow task is mapped to only one service instance. But in computation and data-intensive applications, it is common that the computation resources provided by a single service instance are insufficient for some complicated tasks which contain several closely correlated sub-tasks. To manage such complicated workflows, this article devises a novel workflow model with composite tasks (cWFS). The model views a complicated task as a composite task and allows mapping multiple service instances to a composite task. The data transmission among sub-tasks of a composite task can also be addressed by the proposed model. To solve cWFS problem, we devise a nested particle swarm optimization (N-PSO) that utilizes two kinds of populations, i.e., the outer population and inner population. Since N-PSO is a bit time-consuming, we further devise a Fast version of N-PSO (FN-PSO), which can save more than 60 percent of running time compared with N-PSO. The proposed approaches are evaluated on five real-world workflow types. The experimental results verify that the proposed approaches can solve the new workflow model effectively.

NSE Stock Monitoring using Robotic Process Automation

The core objective of the project is to automate the backend office work of monitoring stocks daily. The project uses “Robotic Process Automation” to download the daily stock values from NSE website, feed the data into an Excel Sheet and send it to the required recipient through E-mail. Software for robotic process automation (RPA) is designed to perform basic tasks across applications just like human workers. A process with multiple steps and applications is taught to the software robot, such as taking received forms, sending a receipt note, verifying the completeness of the document, uploading the form in a folder, and updating a spreadsheet with the form name, the date submitted, etc. Like deep learning, with some support from programmers, the software robots used in robotic process automation are programmed by the employees to do the tasks in a specific workflow. The software is not learning on its own or trying to adapt new efficiencies or new insights such as tools for enterprise resource management (ERM). RPA acts as a remote worker assistant by clearing up the burdensome, simple tasks that eat up part of the day of each office worker.

Deep learning interpretation of echocardiograms

Echocardiography uses ultrasound technology to capture high temporal and spatial resolution images of the heart and surrounding structures, and is the most common imaging modality in cardiovascular medicine. Using convolutional neural networks on a large new dataset, we show that deep learning applied to echocardiography can identify local cardiac structures, estimate cardiac function, and predict systemic phenotypes that modify cardiovascular risk but not readily identifiable to human interpretation. Our deep learning model, EchoNet, accurately identified the presence of pacemaker leads (AUC = 0.89), enlarged left atrium (AUC = 0.86), left ventricular hypertrophy (AUC = 0.75), left ventricular end systolic and diastolic volumes ({R}^{2} = 0.74 and {R}^{2} = 0.70), and ejection fraction ({R}^{2} = 0.50), as well as predicted systemic phenotypes of age ({R}^{2} = 0.46), sex (AUC = 0.88), weight ({R}^{2} = 0.56), and height ({R}^{2} = 0.33). Interpretation analysis validates that EchoNet shows appropriate attention to key cardiac structures when performing human-explainable tasks and highlights hypothesis-generating regions of interest when predicting systemic phenotypes difficult for human interpretation. Machine learning on echocardiography images can streamline repetitive tasks in the clinical workflow, provide preliminary interpretation in areas with insufficient qualified cardiologists, and predict phenotypes challenging for human evaluation.

An effective scheduling strategy based on hypergraph partition in geographically distributed datacenters

As the complexity of workflow applications increase, the scheduling and execution of workflow incur more waste of resources. In order to achieve load balancing and reduce the task execution time in the cloud system, an effective scheduling strategy based on hypergraph partition for workflow application in the geographically distributed datacenters is proposed. Firstly, a workflow job scheduling algorithm is proposed, which aims to reduce the response time and considers the cloud state. Besides, the task scheduling algorithm based on the hypergraph partition is designed with the goal of reducing the completion time and energy consumption for the tasks. In addition, the optimal task scheduling strategy can be obtained according to the Dijkstra shortest path algorithm based on Fibonacci heap. Finally, in the experiment, the workflow task scheduling algorithm can optimize the task execution performance and maintain the load balance of the computing nodes in each cloud, so that the average execution time of the tasks and the total energy consumption of the system are minimized.

AN EFFICIENT FAULT TOLERANT CLUSTERING FOR SCIENTIFIC WORKFLOW

An effective method for the reduction of execution overhead and for improving the computational granularity of scientific workflow tasks that are executing on distributed resources is Task clustering. A job is composed of many tasks and may have a higher risk of suffering from failures than in executing a single task job. In this paper, we direct a hypothetical investigation of the effect of transient failures on the runtime execution of logical work process executions .This system proposes a maximum likelihood estimation-based parameter algorithm which is used for a general task failure modeling framework to model the workflow performance. In this paper, the system proposed here is Dynamic Balanced clustering method which combines the methods of vertical clustering, horizontal clustering and dynamic clustering to reduce the execution overhead for the scientific workflow task execution.

Predicting Workflow Task Execution Time in the Cloud Using A Two-Stage Machine Learning Approach

Many techniques such as scheduling and resource provisioning rely on performance prediction of workflow tasks for varying input data. However, such estimates are difficult to generate in the cloud. This paper introduces a novel two-stage machine learning approach for predicting workflow task execution times for varying input data in the cloud. In order to achieve high accuracy predictions, our approach relies on parameters reflecting runtime information and two stages of predictions. Empirical results for four real world workflow applications and several commercial cloud providers demonstrate that our approach outperforms existing prediction methods. In our experiments, our approach respectively achieves a best-case and worst-case estimation error of 1.6 and 12.2 percent, while existing methods achieved errors beyond 20 percent (for some cases even over 50 percent) in more than 75 percent of the evaluated workflow tasks. In addition, we show that the models predicted by our approach for a specific cloud can be ported with low effort to new clouds with low errors by requiring only a small number of executions.

Minimizing Energy Consumption Scheduling Algorithm of Workflows With Cost Budget Constraint on Heterogeneous Cloud Computing Systems

Cloud computing is a promising platform to conduct large-scale workflow applications according to the pay-per-use model. Minimizing the energy consumption of precedence constrained workflows with cost budget constraints has become one of the popular topics in cloud data centers. Most existing scheduling algorithms mainly consider execution time or cost of a given workflow application under a budget constraint; however, these algorithms do not adequately consider energy saving. A reducing energy consumption strategy using a critical task remapping (RMREC) algorithm is proposed in this study. This algorithm is decomposed into two phases: energy consumption reduction and critical task remapping. In the first phase, the adjustable cost budget and spare cost are determined on the basis of cost budget, critical task path, and adjustable budget factor. All workflow tasks are further allocated to virtual machines (VMs) with the lowest energy consumption to achieve preliminary mapping between tasks and VMs while satisfying the adjustable cost budget constraint. In the second phase, critical tasks are remapped to VMs according to spare cost to decrease energy consumption caused by task migration. Experiments on two types of workflow applications with different scales demonstrate that the presented RMREC algorithm effectively reduces energy consumption without violating cost budget constraints compared with existing algorithms.

A Fast and Accurate Algorithm for Nuclei Instance Segmentation in Microscopy Images

Nuclei instance segmentation within microscopy images is a fundamental task in the pathology work-flow, based on that the meaningful nuclear features can be extracted and multiple biological related analysis can be performed. However, this task is still challenging because of the large variability among different types of nuclei. Although deep learning(DL) based methods have achieved state-of-the-art results in nuclei instance segmentation tasks, these methods are usually focus on improving the accuracy and require support of powerful computing resources. In this paper, we joint the detection and segmentation simultaneously, and propose a fast and accurate box-based nuclei instance segmentation method. Mainly, we employ a fusion module based on the feature pyramid network(FPN) to combine the complementary information of the shallow layers with deep layers for detection the nuclear location by bounding boxes. Subsequently, we crop the feature maps according to the bounding boxes and feed the cropped patches into an U-net architecture as a guide to separate clustered nuclei. The experiments show that the proposed approach outperforms prior state-of-the-art methods, not only on accuracy but also on speed. The source code will be released at: https://github.com/QUAPNH/Nucleiseg.

OBWD: an ontology and Bayesian network-based workflow design platform

Workflow management provides a great convenience for the cooperation between different roles in modern industry and business. The task reuse and design automation are challenges of workflow management currently. In this paper, an ontology SDWMO is constructed for workflow resources integration and task request release. An algorithm DOMDM is proposed to achieve the conversion of the data from traditional workflow data base to SDWMO ontology. In order to provide workflow templates for designers, we extract statistic-oriented cases from the workflow database. Based on these cases a Bayesian network is established for workflow template recommendation. We have designed OBWD platform to implement the above methods. The experimental data indicates that OBWD is statistically effective and saves a lot of time for workflow designers. Currently, OBWD has been used in space debris mitigation domain for workflow management. Moreover, our methodology can also be applied in many other domains in the future.

Формирование механизма управления промышленными предприятиями в условиях ограничительных мер

The article presents current problems faced by industrial enterprises related to the impact of the coronavirus epidemic. Difficulties arising from the production of industrial products, which are directly related to restrictive measures, are disclosed. The key directions of companies in the field of processing are defined, namely: creation of a corporate information management system, improvement of the cash management system, introduction of such mechanisms as” implementation of remote office “and” promotion of production automation. Currently, many manufacturing companies have urgent needs for remote collaborative task management, project management, workflow management, and other software. At the same time, they pay more attention to the practical application of BI, big data analysis, and artificial intelligence. The key advantages of processes for automation and intelligent modernization of fixed assets are identified. It is proved that integration of remote control, intelligent equipment and a high level of automation into production and management processes of industrial enterprises will contribute to development in the short and long term. The main task of the presented research is to determine the key features and relationships of the elements of the management mechanism of industrial enterprises among themselves. At the same time, it was possible to form a conceptual management model based on them. To do this, it is necessary to determine the role of the industrial enterprise management mechanism in the management system. The General structural model of the industrial enterprise management mechanism in the conditions of remote control of divisions is formed. The main goal of integrating the management mechanism of an industrial enterprise is to increase competitive advantages and increase profits

Amber

A long-running analytic task on big data often leaves a developer in the dark without providing valuable feedback about the status of the execution. In addition, a failed job that needs to restart from scratch can waste earlier computing resources. An effective method to address these issues is to allow the developer to debug the task during its execution, which is unfortunately not supported by existing big data solutions. In this paper we develop a system called Amber that supports responsive debugging during the execution of a workflow task. After starting the execution, the developer can pause the job at will, investigate the states of the cluster, modify the job, and resume the computation. She can also set conditional breakpoints to pause the execution when certain conditions are satisfied. In this way, the developer can gain a much better understanding of the run-time behavior of the execution and more easily identify issues in the job or data. Amber is based on the actor model, a distributed computing paradigm that provides concurrent units of computation using actors. We give a full specification of Amber, and implement it on top of the Orleans system. Our experiments show its high performance and usability of debugging on computing clusters.

RPMInter-work: a multi-agent approach for planning the task-role assignments in inter-organisational workflow

ABSTRACTWorkflow management is a core component of modern Enterprise Information Systems (EISs) infrastructure that automates the execution of critical business processes. One of the particular interests of the security community is how to ensure the completion of the workflow execution in the presence of authorisation constraints. These constraints present some restrictions on the users or the roles that are authorised to execute the workflow tasks. The goal is to enforce the legal assignments of access privileges to the executors of the workflow tasks. Despite the variety of approaches proposed in this context, an approach dedicated to the inter-organisational workflows is still missing. In this paper, we take a step towards this goal by proposing a multi-agent-based model, named RPMInter-Work (task-Role assignment Planning Model for Inter-organisational Workflow). Our approach aims to perform the planning of the task-role assignments in inter-organisational workflow in presence of authorisation constraints that are related to task-role assignments. In our research work, this planning problem is formulated as a DisCSP (Distributed Constraint Satisfaction Problem). Our proposed contribution is based on the requirements of inter-organisational workflows, in particular, the autonomy of the participating organisations and the respect of their privacy. A prototype of RPMInter-Work is implemented using JADE (Java Agent DEvelopment) platform and some evaluation results of this prototype are exposed in this paper.