Workflow Execution Research Articles

WOMBAT-P: Benchmarking Label-Free Proteomics Data Analysis Workflows.

The inherent diversity of approaches in proteomics research has led to a wide range of software solutions for data analysis. These software solutions encompass multiple tools, each employing different algorithms for various tasks such as peptide-spectrum matching, protein inference, quantification, statistical analysis, and visualization. To enable an unbiased comparison of commonly used bottom-up label-free proteomics workflows, we introduce WOMBAT-P, a versatile platform designed for automated benchmarking and comparison. WOMBAT-P simplifies the processing of public data by utilizing the sample and data relationship format for proteomics (SDRF-Proteomics) as input. This feature streamlines the analysis of annotated local or public ProteomeXchange data sets, promoting efficient comparisons among diverse outputs. Through an evaluation using experimental ground truth data and a realistic biological data set, we uncover significant disparities and a limited overlap in the quantified proteins. WOMBAT-P not only enables rapid execution and seamless comparison of workflows but also provides valuable insights into the capabilities of different software solutions. These benchmarking metrics are a valuable resource for researchers in selecting the most suitable workflow for their specific data sets. The modular architecture of WOMBAT-P promotes extensibility and customization. The software is available at https://github.com/wombat-p/WOMBAT-Pipelines.

Optimization of Maritime Communication Workflow Execution with a Task-Oriented Scheduling Framework in Cloud Computing

To ensure safe, effective, and efficient marine operations, the optimization of maritime communication workflows with a task-oriented scheduling framework is of the utmost importance. Navigation, vessel traffic management, emergency response, and cargo operations are all made possible by maritime communication, which necessitates seamless information sharing between ships, ports, coast guards, and regulatory bodies. However, traditional communication methods face challenges in adapting to the dynamic and distributed nature of maritime activities. This study suggests a novel approach for overcoming these difficulties that combines task-oriented scheduling and resource-aware cloud environments to enhance marine communication operations. Utilizing cloud computing offers a scalable, adaptable infrastructure that can manage various computational and communication needs. Even during busy times, effective data processing, improved decision making, and improved communication are made possible by utilizing the cloud. The intelligent allocation and prioritization of communication activities using a task-oriented scheduling framework ensures that urgent messages receive prompt attention while maximizing resource utilization. The proposed approach attempts to improve marine communication workflows’ task prioritization, scalability, and resource optimization. In order to show the effectiveness of the proposed approach, simulations were performed in CloudSim. The performance evaluation parameters, i.e., throughput, latency, execution cost, and energy consumption, have been evaluated. Simulation results reflect the efficacy and practical usability of the framework in various maritime communication configurations. By making marine communication methods more durable, dependable, and adaptable to the changing needs of the maritime industry, this study advances maritime communication techniques. The findings of this research have the potential to revolutionize maritime communication, leading to safer, more efficient, and more resilient maritime operations on a large scale.

Analysing methods of Workflow Execution on Data Mining

Analysing methods of Workflow Execution on Data Mining

Multitasking bi-level evolutionary algorithm for data-intensive scientific workflows on clouds

With the deployment of workflow and other applications, cloud computing is accessible and offers assistance for optimizing workflow execution and enhancing performance. Existing research, however, tends to disregard the influence of dataset migration on workflow execution and focuses more on task execution time. This study suggests a new model for the problem of data-intensive workflow execution. Firstly, according to the structure of the workflow scheduling problem, it is divided into two sub-problems: data placement and task scheduling. The two sub-problems interact with each other and a bi-level optimum model is established. By seeking a better allocation strategy for the dataset placement and then seeking the best task-scheduling solution. Secondly, an improved multitasking bi-level evolutionary algorithm (IM-BLEA) is proposed. When dealing with the lower-level optimization problem (LLOP), offspring are selected by sorting individuals by their performance and overall performance in the population, and this environmental selection enhances the diversity and searchability of the population. Finally, compared with the other multitasking algorithm, IM-BLEA has good performance. Simulation results based on real scientific workflows show that the algorithm improves the values of transfer time and number of selected data centers by 56% and 10% compared to the comparison algorithm.

Nucleic acid based point-of-care diagnostic technology for infectious disease detection using machine learning empowered smartphone-interfaced quantitative colorimetry

We report a nucleic acid-based point of care testing technology for infectious disease detection at resource limited settings by integrating a low-cost portable device with machine learning-empowered quantitative colorimetric analytics that can be interfaced via a smartphone application. We substantiate our proposition by demonstrating the efficacy of this technology in detecting COVID-19 infection from human swab samples, using the RT-LAMP protocol. Comparison with gold standard results from real-time PCR evidences high sensitivity and specificity, ensuring simplicity, portability, and user-friendliness of the technology at the same time. Colorimetric analytics of the reaction output without necessitating the opening of the reaction microchambers enables execution of the complete test workflow without any laboratory control that may otherwise be required stringently for safeguarding against carryover contamination. Seamless sample-to-answer workflow and machine learning-based readout further assures minimal human intervention for the test readout, thus eliminating inevitable inaccuracies stemming from erroneous execution of the test as well as subjectivity in interpreting the outcome. Our results further indicate the possibilities of upgrading the technology to predict the pathogenic load on the infected patients akin to the cyclic threshold value of the real-time PCR, when calibrated with reference to a wide range of ‘training’ data for the machine learner, thereby putting forward the same as viable alternative to the resource-intensive PCR tests that cannot be made readily accessible at underserved community settings.

SCAFE: A Service-Centered Cloud-Native Workflow Engine Architecture

With the rapid development of manufacturing and cloud computing, more and more emerged cloud services provide a promising way to perform complex requirements efficiently. Workflow offers an effective way to assemble disparate services and interacts with them to construct business logic for user requests. Meanwhile, workflow engines are responsible for the control of workflow execution. However, existing engines usually support interaction between workflows and services or computing resources by tight binding approaches, which lack flexibility and scalability. Therefore, a flexible and decoupled architecture is essential to support automatic workflow management. To fill this gap, this paper proposes a novel <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">s</b> ervice- <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">c</b> entered cloud-n <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a</b> tive work <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f</b> low architectur <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e</b> - SCAFE. The introduction of the service layer in SCAFE decouples the upper business services and lower execution resources, facilitating the independent and joint management of three execution objects (request - service - execution instance) involved in the cloud workflow lifecycle. We present a 2-stage scheduling model for the new architecture to support customized service optimization and resource-aware task scheduling. In addition, a fault-tolerant mechanism is integrated into resolving task execution exceptions quickly. We have successfully implemented a prototype tool to verify its flexibility and crucial functions, thus advancing the field workflow engines in cloud-native environments.

Data-intensive workflow scheduling strategy based on deep reinforcement learning in multi-clouds

With the increase development of Internet of Things devices, the data-intensive workflow has emerged as a new kinds of representation for IoT applications. Because most IoT systems are structured in multi-clouds environment and the data-intensive workflow has the characteristics of scattered data sources and distributed execution requirements at the cloud center and edge clouds, it brings many challenges to the scheduling of such workflow, such as data flow control management, data transmission scheduling, etc. Aiming at the execution constraints of business and technology and data transmission optimization of data-intensive workflow, a data-intensive workflow scheduling method based on deep reinforcement learning in multi-clouds is proposed. First, the execution constraints, edge node load and data transmission volume of IoT data workflow are modeled; then the data-intensive workflow is segmented with the consideration of business constraints and the first optimization goal of data transmission; besides, taking the workflow execution time and average load balancing as the secondary optimization goal, the improved DQN algorithm is used to schedule the workflow. Based on the DQN algorithm, the model reward function and action selection are redesigned and improved. The simulation results based on WorkflowSim show that, compared with MOPSO, NSGA-II, GTBGA and DQN, the algorithm proposed in this paper can effectively reduce the execution time of IoT data workflow under the condition of ensuring the execution constraints and load balancing of multi-clouds.

SWEL: A Domain-Specific Language for Modeling Data-Intensive Workflows

AbstractData-intensive applications aim at discovering valuable knowledge from large amounts of data coming from real-world sources. Typically, workflow languages are used to specify these applications, and their associated engines enable the execution of the specifications. However, as these applications become commonplace, new challenges arise. Existing workflow languages are normally platform-specific, which severely hinders their interoperability with other languages and execution engines. This also limits their reusability outside the platforms for which they were originally defined. Following the Design Science Research methodology, the paper presents SWEL (Scientific Workflow Execution Language). SWEL is a domain-specific modeling language for the specification of data-intensive workflows that follow the model-driven engineering principles, covering the high-level definition of tasks, information sources, platform requirements, and mappings to the target technologies. SWEL is platform-independent, enables collaboration among data scientists across multiple domains and facilitates interoperability. The evaluation results show that SWEL is suitable enough to represent the concepts and mechanisms of commonly used data-intensive workflows. Moreover, SWEL facilitates the development of related technologies such as editors, tools for exchanging knowledge assets between workflow management systems, and tools for collaborative workflow development.

Building a Collaborative Data Analytics System: Opportunities and Challenges

Real-time collaboration has become increasingly important in various applications, from document creation to data analytics. Although collaboration features are prevalent in editing applications, they remain rare in data-analytics applications, where the need for collaboration is even more crucial. This tutorial aims to provide attendees with a comprehensive understanding of the challenges and design decisions associated with supporting real-time collaboration and user interactions in data analytics systems. We will discuss popular conflict resolution technologies, the unique challenges of facilitating collaborative experiences during the workflow construction and execution phases, and the complexities of supporting responsive user interactions during job execution.

Knowledge-driven adaptive evolutionary multi-objective scheduling algorithm for cloud workflows

Workflow scheduling in cloud platforms is a highly challenging issue because it faces multiple conflicting optimization objectives and large-scale decision variables. Most of the existing multi-objective workflow scheduling algorithms regard the focused problems as black boxes, and optimize large-scale decision variables as a whole. This leads to inefficiency in searching solution spaces that grow exponentially with the increase of decision variables. To compensate the above deficiency, this paper proposes a knowledge-driven adaptive evolutionary multi-objective scheduling algorithm, KAMSA for short, to optimize makespan and cost of workflow execution in cloud platforms. Specifically, we excavate the knowledge that adjustment of a task’s execution only affects its successor tasks to divide large-scale decision variables into a series of groups, so as to give play to the strengths of divide-and-conquer technology to improve the evolutionary search efficiency. Moreover, we develop an adaptive resource allocation scheme to reward more evolution opportunities for groups with high contributions to further improve the evolutionary search efficiency. We compare the proposed KAMSA with five state-of-the-art competitors in the context of 20 real-world workflows and the Amazon elastic compute cloud (EC2). The comparison results verify the KAMSA’s advantages by prevailing over the five competitors on 18 out of the 20 test cases with respect to the metric hypervolume.

A Fully Hybrid Algorithm for Deadline Constrained Workflow Scheduling in Clouds

With the migration of more and more workflows to clouds, the workflow scheduling in clouds (WSC) becomes a critical problem. Although many algorithms have been presented for WSC, there is still room and need for improvement. This paper formulates WSC as a constrained optimization problem that optimizes workflow execution cost within a workflow deadline constraint and proposes a fully hybrid workflow scheduling algorithm, called HPCP-PSO to solve it. Unlike previous works, HPCP-PSO is based on the repeated and alternated execution of two different methods, namely, the heuristic IaaS Cloud Partial Critical Paths (IC-PCP) and meta-heuristic Particle Swarm Optimization (PSO). Moreover, HPCP-PSO incorporates with two novel designs: 1) a new solution encoding strategy not only to sufficiently embody the elasticity of cloud resources, but also to reflect the scheduling relationship between assigned and unassigned tasks; 2) a solution repair strategy on each infeasible lease process to utilize a user-defined deadline more effectively and enhance the solution efficiency of the algorithm. Extensive experiments are conducted on four real-world scientific workflows and the results show that compared with IC-PCP, PSO, and HGSA, the proposed algorithm outperforms them on average by 35.83%, 70.53%, and 87.71% in terms of workflow execution cost.

Research on Lightweight Microservice Composition Technology in Cloud-Edge Device Scenarios

In recent years, cloud-native technology has become popular among Internet companies. Microservice architecture solves the complexity problem for multiple service methods by decomposing a single application so that each service can be independently developed, independently deployed, and independently expanded. At the same time, domestic industrial Internet construction is still in its infancy, and small and medium-sized enterprises still face many problems in the process of digital transformation, such as difficult resource integration, complex control equipment workflow, slow development and deployment process, and shortage of operation and maintenance personnel. The existing traditional workflow architecture is mainly aimed at the cloud scenario, which consumes a lot of resources and cannot be used in resource-limited scenarios at the edge. Moreover, traditional workflow is not efficient enough to transfer data and often needs to rely on various storage mechanisms. In this article, a lightweight and efficient workflow architecture is proposed to optimize the defects of these traditional workflows by combining cloud-edge scene. By orchestrating a lightweight workflow engine with a Kubernetes Operator, the architecture can significantly reduce workflow execution time and unify data flow between cloud microservices and edge devices.

Scientific workflow execution in the cloud using a dynamic runtime model

To explain specific phenomena, scientists perform a sequence of tasks, e.g., to gather, analyze and interpret data, forming a scientific workflow. Depending on the complexity of the workflow, scientists require access to various kinds of tools, applications and infrastructures for individual tasks. Current approaches are often limited to managing these resources at design time, requiring the scientist to preemptively set up applications essential for their workflow. Therefore, a dynamic provisioning and configuration of computing resources are required that fulfills these needs at runtime. In this paper, we present a dynamic runtime model that couples workflow tasks with their individual applications and infrastructure requirements. This runtime model is used as a knowledge base by a model-driven workflow execution engine orchestrating the sequence of tasks and their infrastructure. We exhibit that the simplicity of the runtime model supports the creation of highly tailored infrastructures, the integration of self-developed applications, as well as a human-in-the-loop allowing scientists to monitor and interact with the workflow at runtime. To tackle the heterogeneity of cloud provider interfaces, we implement the workflow runtime model by extending the Open Cloud Computing Interface cloud standard, which provides an extensible data model as well as a uniform interface to manage cloud resources. We demonstrate the applicability of our approach using three case studies and discuss the benefits of the runtime model from a user and system perspective.

Graph neural networks for detecting anomalies in scientific workflows

Identifying and addressing anomalies in complex, distributed systems can be challenging for reliable execution of scientific workflows. We model these workflows as directed acyclic graphs (DAGs), where the nodes and edges of the DAGs represent jobs and their dependencies, respectively. We develop graph neural networks (GNNs) to learn patterns in the DAGs and to detect anomalies at the node (job) and graph (workflow) levels. We investigate workflow-specific GNN models that are trained on a particular workflow and workflow-agnostic GNN models that are trained across the workflows. Our GNN models, which incorporate both individual job features and topological information from the workflow, show improved accuracy and efficiency compared to conventional learning methods for detecting anomalies. While joint trained with multiple scientific workflows, our GNN models reached an accuracy more than 80% for workflow level and 75% for job level anomalies. In addition, we illustrate the importance of hyperparameter tuning method in our study that can significantly improve the metric(s) measure of evaluating the GNN models. Finally, we integrate explainable GNN methods to provide insights on job features in the workflow that cause an anomaly.

Redefining the learning mechanism in teaching‐learning‐based optimization and its applications for flowtime‐aware‐cost minimizing of the workflow in cloud

SummaryTeaching‐learning‐based optimization (TLBO) algorithm is a population‐based meta‐heuristic algorithm that was created to solve single‐objective optimization problems. The teaching‐learning mechanism of a classroom inspires it. TLBO suffers from weak exploration. As a result, its performance is not good for solving multimodal problems. To turn TLBO into a tool for solving multimodal problems and maintaining good diversity, we made significant modifications into the learning process of the fundamental TLBO. The proposed algorithm produces more diverse solutions and works better for solving multimodal problems. This newly created variant of TLBO is called “Intelligent‐Teaching‐Learning‐Based Optimization (I‐TLBO) algorithm.” I‐TLBO's performance is evaluated against the most recent standard benchmark function, CEC‐06, 2019, and it is discovered that I‐TLBO outperforms the other algorithms. After that, I‐TLBO was applied for flowtime‐aware‐cost minimization of the workflow executions in cloud datacenter. To solve these scheduling problems, I‐TLBO and other metaheuristic algorithms are simulated in CloudSim and tested over scientific workflows such as Inspiral, Montage, SIPHT, sample, Cybershake, and Epigenomics workflows. Finally, it is found that I‐TLBO reduces flowtime and cost both by 28.48%, 11.30%, 17.64%, 13.22%, 11.45%, and 14.71% in comparison to the second best performing algorithm while executing the standard workflow in cloud.

And synopses for all: A synopses data engine for extreme scale analytics-as-a-service

In this work, we detail the design and structure of a Synopses Data Engine (SDE) which combines the virtues of parallel processing and stream summarization towards delivering interactive analytics at extreme scale. Our SDE is built on top of Apache Flink and implements a novel synopses-as-a-service paradigm. In that, it achieves (i) concurrently maintaining thousands of synopses of various types for thousands of streams, on demand, (ii) reusing synopses that are common across various concurrent workflows, (iii) providing data summarization facilities even for cross-(Big Data) platform workflows, (iv) pluggability of new synopses on-the-fly, (v) increased potential for workflow execution optimization. The proposed SDE-as-a-service provides interactive analytics at scale by enabling 3 types of scalability: (i) enhanced horizontal scalability, i.e., not only scaling out the computation to a number of processing units available in a computer cluster, but also harnessing the processing load assigned to each by operating on carefully-crafted data summaries, (ii) vertical scalability, i.e., scaling the computation to very high numbers of processed streams and (iii) federated scalability i.e., scaling across geo-distributed clusters and clouds by controlling the communication required to answer global queries.

Evolutionary Optimization of Energy Consumption and Makespan of Workflow Execution in Clouds

Making sound trade-offs between the energy consumption and the makespan of workflow execution in cloud platforms remains a significant but challenging issue. So far, some works balance workflows’ energy consumption and makespan by adopting multi-objective evolutionary algorithms, but they often regard this as a black-box problem, resulting in the low efficiency of the evolutionary search. To compensate for the shortcomings of existing works, this paper mathematically formulates the cloud workflow scheduling for an infrastructure-as-a-service (IaaS) platform as a multi-objective optimization problem. Then, this paper tailors a knowledge-driven energy- and makespan-aware workflow scheduling algorithm, namely EMWSA. Specifically, a critical task adjustment-based local search strategy is proposed to intelligently adjust some critical tasks to the same resource of their successor tasks, striving to simultaneously reduce workflows’ energy consumption and makespan. Further, an idle gap reuse strategy is proposed to search the optimal energy consumption of each non-critical task without affecting the operation of other tasks, so as to further reduce energy consumption. Finally, in the context of real-world workflows and cloud platforms, we carry out comparative experiments to verify the superiority of the proposed EMWSA by significantly outperforming 4 representative baselines on 19 out of 20 workflow instances.

Adaptive Cloud Bundle Provisioning and Multi-Workflow Scheduling via Coalition Reinforcement Learning

The efficient cloud resource provisioning for the execution of complex workflow applications has always been one of the important research issues. Most of the existing approaches focus on the resource provisioning of single-type virtual machine (VM) instances for the single or multiple workflows, while few consider the situation of provisioning multi-type VM instances simultaneously. As a result, the executing performance of complex workflows degrades. Different from the existing work, this paper proposes an adaptive cloud bundle provisioning and multi-workflow scheduling model to dynamically perform both the horizontal and vertical cloud resource scaling on multi-type VM instances for the execution of complex workflows. Among the model, a depth-first-search coalition reinforcement learning (DFSCRL) provisioning policy is presented to realize the resource scaling, which integrates the physical machine (PM) coalition formation with the Q-learning algorithm, then dynamically generates an optimal multi-type VM instance bundle from the PM coalition, and finally provisions these instances to the concurrent execution of multiple workflows. The theoretical proofs and various experiments with the multifaceted metrics demonstrate that the performance of the proposed algorithms is superior to that of the state-of-the-art relevant policies.

BUSINESS PLAN : RENCANA PENDIRIAN EVENT ORGANIZER

Opportunities for business services for organizing activities that are increasingly widespread and developing, either as activity planners or activity organizers. Here preparations that must be made to conduct events such as community gathering, music concert, wedding, and corporate events are discussed. The objective of this study is to determine strategies that will be realized based on the analyses on the related aspects in order to develop and to face competition in event organizing business in Bojonegoro. This qualitative descriptive research uses primary data obtained from observations and interviews and secondary data compiled from relevant literature. This study finds that the business planning has met the necessary aspects such as analysis of customer’s interest, support from the government, the availability of other companies for partnership purposes, planning workflow and event execution, analyses on competitors of the same field of industry, required service variations, form of partnership agreements, business legality, and analyses of sponsors from other companies as the supporting parties for the organization of an event.

Provenance-Preserving Analysis and Rewrite of Quantum Workflows for Hybrid Quantum Algorithms

Quantum applications are hybrid, i.e., they comprise quantum and classical programs, which must be orchestrated. Workflows are a proven solution for orchestrating heterogeneous programs while providing benefits, such as robustness or scalability. However, the orchestration using workflows can be inefficient for some quantum algorithms, requiring the execution of quantum and classical programs in a loop. Hybrid runtimes are offered to efficiently execute these algorithms. For this, the quantum and classical programs are combined in a single hybrid program, for which the execution is optimized. However, this leads to a conceptual gap between the modeling benefits of workflow technologies, e.g., modularization, reuse, and understandability, and the efficiency improvements when using hybrid runtimes. To close this gap, we introduce a method to model all tasks explicitly in the workflow model and analyze the workflow to detect parts of the workflow that can benefit from hybrid runtimes. Furthermore, corresponding hybrid programs are automatically generated based on the quantum and classical programs, and the workflow is rewritten to invoke them. To ease the live monitoring and later analysis of workflow executions, we integrate process views into our method and collect related provenance data. Thus, the user can visualize and monitor the workflow in the original and rewritten form within the workflow engine. The practical feasibility of our approach is validated by a prototypical implementation, a case study, and a runtime evaluation.