Workflow Orchestration Research Articles

Streamlining financial investment operations: Leveraging java full stack, angular and AWS cloud technologies for a customizable automated letter printing system

Java Full Stack and Angular and AWS Cloud Technologies drive fundamental transformations in financial investment operations through automated letter printing systems development. This modern operational solution makes processing and distributing customized client correspondence automatic. A Spring Boot and Java-based robust backend handles efficient data processing tasks while ensuring smooth connection with multiple financial data sources. Using RESTful APIs gives users immediate access to real-time data and customization features that enhance system connectivity. Angular's front end reacts to different screen sizes, presenting an easy-to-use interface that enables effortless template and workflow management. The AWS Cloud enables this system through its scalable platform while providing reliable computing power, including Lambda automation, S3 data storage, and Step Functions for workflow orchestration. When implemented, the system helps companies reduce manual work while improving precision, regulatory adherence, and customer satisfaction to advance financial operations efficiently with client focus.

Integration Patterns in Unified AI and Cloud Platforms: A Systematic Review of Process Automation Technologies

This article comprehensively analyzes unified AI and cloud platforms, examining their role in transforming process automation and decision systems across industries. The article investigates the architectural frameworks and integration patterns that enable the convergence of AI tools, machine learning operations, and workflow orchestration within cloud-native environments. The article explores key innovations, including federated AI implementations, real-time data processing architectures, and multi-cloud integration patterns. It provides insights into their practical applications across finance, healthcare, retail, and manufacturing sectors. The article identifies critical success factors in platform implementation, including integrating MLOps frameworks, automated decision engines, and compliance tools for AI governance. Through case study analysis and architectural evaluation, we demonstrate how unified platforms address traditional challenges in AI deployment while enabling scalable, cost-efficient solutions. The findings reveal emerging patterns in platform architecture that facilitate seamless integration of edge computing, real-time analytics, and distributed AI systems, contributing to the broader understanding of enterprise AI implementation strategies. This article provides valuable insights for researchers and practitioners in cloud engineering, artificial intelligence, and systems integration while highlighting future directions for platform evolution and standardization.

Orchestration Workflows in Distributed Systems: A Systematic Analysis of Efficiency Optimization and Service Coordination

This article presents a comprehensive analysis of orchestration workflows in modern software architectures, examining their role in optimizing service efficiency and coordination across distributed systems. Through a systematic investigation of implementation patterns and operational strategies, we demonstrate how orchestrated workflows enhance service integration, automate complex processes, and improve system reliability. The article employs a mixed-method approach, combining quantitative performance metrics from real-world implementations with qualitative analysis of architectural patterns across diverse enterprise environments. The findings reveal that organizations implementing robust orchestration workflows experienced a 47% reduction in service integration failures, 35% improvement in resource utilization, and 62% faster deployment cycles. The article also identifies critical success factors in orchestration implementation, including dynamic resource management, automated error handling, and comprehensive monitoring frameworks. Furthermore, The article proposes a novel framework for evaluating orchestration effectiveness in cloud-native environments, considering factors such as service mesh integration, container orchestration, and distributed tracing. This article contributes to the growing body of knowledge in service optimization by providing empirical evidence of orchestration benefits and offering practical guidelines for implementing scalable, resilient software systems through advanced workflow orchestration strategies.

Cloud-Native ETL: Integrating Databricks and Azure Data Factory for Scalable Data Processing in Enterprise Environments

This article examines the implementation of cloud-native ETL solutions leveraging Databricks and Azure Data Factory (ADF) for scalable data processing in enterprise environments. The article presents a comprehensive analysis of the architectural design, integration strategies, and performance optimization techniques for combining Databricks' powerful data transformation capabilities with ADF's robust workflow orchestration. Through a series of case studies and empirical evaluations, we demonstrate how this integrated approach addresses the challenges of big data processing, including scalability, flexibility, and cost-effectiveness. Our findings reveal significant improvements in processing efficiency and resource utilization, with observed reductions in ETL pipeline execution times by up to 40% and overall cloud infrastructure costs by 25%. The article also highlights best practices for data governance, security, and quality management within this framework. These insights provide valuable guidance for data engineers and IT professionals seeking to modernize their data processing infrastructure and harness the full potential of cloud-native ETL solutions.

An empirical study of developers’ challenges in implementing Workflows as Code: A case study on Apache Airflow

The Workflows as Code paradigm is becoming increasingly essential to streamline the design and management of complex processes within data-intensive software systems. These systems require robust capabilities to process, analyze, and extract insights from large datasets. Workflow orchestration platforms such as Apache Airflow are pivotal in meeting these needs, as they effectively support the implementation of the Workflows as Code paradigm. Nevertheless, despite its considerable advantages, developers still face challenges due to the specialized demands of workflow orchestration and the complexities of distributed execution environments. In this paper, we manually study 1000 sampled Stack Overflow posts derived from 9591 Airflow-related questions to understand developers’ challenges and root causes while implementing Workflows as Code. Our analysis results in a hierarchical taxonomy of Airflow-related challenges that contains 7 high-level categories and 14 sub-categories. We find that the most significant obstacles for developers arise when defining and executing their workflow. Our in-depth analysis identifies 10 root causes behind the challenges, including incorrect workflow configuration, complex environmental setup, and a lack of basic knowledge about Airflow and the external systems that it interacts with. Additionally, our analysis of references shared within the collected posts reveals that beyond the frequently cited Airflow documentation, documentation from external systems and third-party providers is also commonly referenced to address Airflow-related challenges.Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

A Demonstration of the Risk-Informed NEI 18-04 Design Evaluation Model for the Modular High-Temperature Gas Reactor

Several advanced reactor designs are now under active development in the United States and elsewhere, promising sustainable solutions to the growing world energy needs. The designs currently being considered are quite diverse and different from the more established light water reactor technology that has dominated the operating commercial nuclear landscape. While advanced reactor concepts were first explored in the dawn of the nuclear age, they are now being reconsidered under the light of modern needs, and specifically, for their flexible operating conditions and inherent safety characteristics. In response, the U.S. Nuclear Regulatory Commission staff is moving forward with development of 10 CFR Part 53 rulemaking, which is a more risk-informed, technology-agnostic framework for licensing and regulating such new designs. The nuclear industry response to this regulatory initiative resulted in the technical report by the Nuclear Energy Institute, NEI 18-04 Revision 1, which provides an implementation roadmap of the risk-informed approach when defining the safety case for a new plant design. The implementation of this safety case may be a nontrivial exercise for an actual reactor design. This paper provides a demonstration of performing such an analysis for a representative advanced reactor. Public information from the General Atomics high-temperature gas reactor design was considered in this demonstration. The analysis workflow was facilitated with the FPoliSolutions’ proprietary Risk-Informed System Engineering (RISE) digital platform, a product that was presented in previous publications. RISE is one application of FPoli’s enterprise digital platform, which was created to facilitate orchestration of complex workflows leveraging recent technologies developed at national laboratories, such as Idaho National Laboratory’s RAVEN and EMRALD frameworks. The analysis described in the paper includes the selection and classifications of events, the integration of probabilistic risk analysis artifacts, and event modeling simulations for consequence evaluations. The results are then used for system, structures, and components safety classification and a synthesis of the safety case for the design in line with the frequency-consequence targets presented in NEI 18-04. The purpose of the analysis, as framed in RISE, is to readily produce outputs and views that can aid users and regulators in making risk-informed decisions to demonstrate their plant safety case.

Deep integration of low-cost liquid handling robots in an industrial pharmaceutical development environment

The pharmaceutical industry is increasingly embracing laboratory automation to enhance experimental efficiency and operational resilience, particularly through the integration of automated liquid handlers (ALHs). This paper explores the integration of the low-cost Opentrons OT-2 liquid handling robot with F. Hoffmann-La Roche AG's in-house workflow orchestration software, AutoLab, to overcome barriers to lab automation. By leveraging the OT-2′s development-oriented interfaces and AutoLab's modular architecture, we achieved a user-friendly, cost-efficient, and flexible automation solution that aligns with FAIR (findable, accessible, interoperable, reusable) data principles. We demonstrate an advanced workflow development methodology, utilizing the software architecture, that facilitates the creation of two flexible pipetting protocols and medium complexity assays. This deep integration approach diminishes the learning curve for novice users while simultaneously enhancing the overall efficiency and reliability of the experimental workflow. Our findings suggest that such integrations can significantly mitigate the challenges associated with lab automation, including cost, complexity, and adaptability, paving the way for more accessible and robust automated systems in pharmaceutical research.

AssemblyQC: a Nextflow pipeline for reproducible reporting of assembly quality.

Genome assembly projects have grown exponentially due to breakthroughs in sequencing technologies and assembly algorithms. Evaluating the quality of genome assemblies is critical to ensure the reliability of downstream analysis and interpretation. To fulfil this task, we have developed the AssemblyQC pipeline that performs file-format validation, contaminant checking, contiguity measurement, gene- and repeat-space completeness quantification, telomere inspection, taxonomic assignment, synteny alignment, scaffold examination through Hi-C contact-map visualization, and assessments of completeness, consensus quality and phasing through k-mer analysis. It produces a comprehensive HTML report with method descriptions, tables, and visualizations. The pipeline uses Nextflow for workflow orchestration and adheres to the best-practice established by the nf-core community. This pipeline offers a reproducible, scalable, and portable method to assess the quality of genome assemblies-the code is available online at GitHub: https://github.com/Plant-Food-Research-Open/assemblyqc.

Data pipeline approaches in serverless computing: a taxonomy, review, and research trends

Serverless computing has gained significant popularity due to its scalability, cost-effectiveness, and ease of deployment. With the exponential growth of data, organizations face the challenge of efficiently processing and analyzing vast amounts of data in a serverless environment. Data pipelines play a crucial role in managing and transforming data within serverless architectures. This paper provides a taxonomy of data pipeline approaches in serverless computing. Classification is based on architectural features, data processing techniques, and workflow orchestration mechanisms, these approaches are categorized into three primary methods: heuristic-based approach, Machine learning-based approach, and framework-based approach. Furthermore, a systematic review of existing data pipeline frameworks and tools is provided, encompassing their strengths, limitations, and real-world use cases. The advantages and disadvantages of each approach, also the challenges and performance metrics that influence their effectuality have been examined. Every data pipeline approach has certain advantages and disadvantages, whether it is framework-based, heuristic-based, or machine learning-based. Each approach is suitable for specific use cases. Hence, it is crucial assess the trade-offs between complexity, performance, cost, and scalability, while selecting a data pipeline approach. In the end, the paper highlights a number of open issues and future investigations directions for data pipeline in the serverless computing, which involve scalability, fault tolerance, data real time processing, data workflow orchestration, function state management with performance and cost in the serverless computing environments.

Harmonizing Compliance: Coordinating Automated Verification Processes within Cloud-based AI/ML Workflows

The significance of ensuring security and upholding data privacy within cloud-based workflows is widely recognized in research domains. This importance is particularly evident in contexts such as safeguarding patients' private data managed within cloud-deployed workflows, where maintaining confidentiality is paramount, alongside ensuring secure communication among involved stakeholders. In response to these imperatives, our paper presents an architecture and formal model designed to enforce security measures within cloud workflow orchestration. Central to our proposed architecture is the emphasis on continuous monitoring of cloud resources, workflow tasks, and data streams to detect and preempt anomalies in workflow orchestration processes. To accomplish this, we advocate for a multi-modal approach that integrates deep learning, one-class classification, and clustering techniques. In essence, our proposed architecture offers a comprehensive solution for enforcing security within cloud workflow orchestration, harnessing advanced methodologies like deep learning for anomaly detection and prediction. This approach is particularly pertinent in critical sectors such as healthcare, especially during unprecedented events like the COVID-19 pandemic.

Harmonizing Compliance: Coordinating Automated Verification Processes within Cloud-based AI/ML Workflows

The significance of ensuring security and upholding data privacy within cloud-based workflows is widely recognized in research domains. This importance is particularly evident in contexts such as safeguarding patients' private data managed within cloud-deployed workflows, where maintaining confidentiality is paramount, alongside ensuring secure communication among involved stakeholders. In response to these imperatives, our paper presents an architecture and formal model designed to enforce security measures within cloud workflow orchestration. Central to our proposed architecture is the emphasis on continuous monitoring of cloud resources, workflow tasks, and data streams to detect and preempt anomalies in workflow orchestration processes. To accomplish this, we advocate for a multi-modal approach that integrates deep learning, one-class classification, and clustering techniques. In essence, our proposed architecture offers a comprehensive solution for enforcing security within cloud workflow orchestration, harnessing advanced methodologies like deep learning for anomaly detection and prediction. This approach is particularly pertinent in critical sectors such as healthcare, especially during unprecedented events like the COVID-19 pandemic.

Data and information management for acoustics research

A critical part of verification and validation in academic research includes the important consideration of data and information management. As researchers grapple with escalating volumes of data, effective data management becomes imperative for optimizing operational processes and ensuring reproducibility and archivability. Data management involves the organization, storage, and retrieval of information to support research advancements and strengthen the foundation of decision making and scientific knowledge. Key components also include data operations, which involves the orchestration of data workflows, and data quality management, which focuses on maintaining accurate and reliable data. This talk explores the multifaceted aspects of data management, emphasizing its significance in ensuring data quality, accuracy and accessibility.

AlabOS: a Python-based reconfigurable workflow management framework for autonomous laboratories

AlabOS is a workflow orchestration framework designed to address the increased complexity in autonomous laboratories, featuring a reconfigurable experiment workflow model and a resource reservation mechanism.

Streamlining Compliance: Orchestrating Automated Checks for Cloud-based AI/ML Workflows

Ensuring security and safeguarding data privacy within cloud workflows has garnered considerable attention in research circles. For instance, protecting the confidentiality of patients' private data managed within a cloud-deployed workflow is crucial, as is ensuring secure communication of such sensitive information among various stakeholders. In light of this, our paper proposes an architecture and a formal model for enforcing security within cloud workflow orchestration. The proposed architecture underscores the importance of monitoring cloud resources, workflow tasks, and data to identify and anticipate anomalies in cloud workflow orchestration. To achieve this, we advocate a multi-modal approach combining deep learning, one-class classification, and clustering techniques. In summary, our proposed architecture offers a comprehensive solution to security enforcement within cloud workflow orchestration, leveraging advanced techniques like deep learning for anomaly detection and prediction, particularly pertinent in critical domains such as healthcare during unprecedented times like the COVID-19 pandemic.

Digital transformation of career landscapes in radiology: personal and professional implications.

Millennial radiology is marked by technical disruptions. Advances in internet, digital communications and computing technology, paved way for digitalized workflow orchestration of busy radiology departments. The COVID pandemic brought teleradiology to the forefront, highlighting its importance in maintaining continuity of radiological services, making it an integral component of the radiology practice. Increasing computing power and integrated multimodal data are driving incorporation of artificial intelligence at various stages of the radiology image and reporting cycle. These have and will continue to transform the career landscape in radiology, with more options for radiologists with varied interests and career goals. The ability to work from anywhere and anytime needs to be balanced with other aspects of life. Robust communication, internal and external collaboration, self-discipline, and self-motivation are key to achieving the desired balance while practicing radiology the unconventional way.

Orchestration of materials science workflows for heterogeneous resources at large scale

In the era of big data, materials science workflows need to handle large-scale data distribution, storage, and computation. Any of these areas can become a performance bottleneck. We present a framework for analyzing internal material structures (e.g., cracks) to mitigate these bottlenecks. We demonstrate the effectiveness of our framework for a workflow performing synchrotron X-ray computed tomography reconstruction and segmentation of a silica-based structure. Our framework provides a cloud-based, cutting-edge solution to challenges such as growing intermediate and output data and heavy resource demands during image reconstruction and segmentation. Specifically, our framework efficiently manages data storage, scaling up compute resources on the cloud. The multi-layer software structure of our framework includes three layers. A top layer uses Jupyter notebooks and serves as the user interface. A middle layer uses Ansible for resource deployment and managing the execution environment. A low layer is dedicated to resource management and provides resource management and job scheduling on heterogeneous nodes (i.e., GPU and CPU). At the core of this layer, Kubernetes supports resource management, and Dask enables large-scale job scheduling for heterogeneous resources. The broader impact of our work is four-fold: through our framework, we hide the complexity of the cloud’s software stack to the user who otherwise is required to have expertise in cloud technologies; we manage job scheduling efficiently and in a scalable manner; we enable resource elasticity and workflow orchestration at a large scale; and we facilitate moving the study of nonporous structures, which has wide applications in engineering and scientific fields, to the cloud. While we demonstrate the capability of our framework for a specific materials science application, it can be adapted for other applications and domains because of its modular, multi-layer architecture.

SGMFQP: An ontology-based Swine Gut Microbiota Federated Query Platform.

Gut microbiota plays a crucial role in modulating pig development and health, and gut microbiota characteristics are associated with differences in feed efficiency. To answer open questions in feed efficiency analysis, biologists seek to retrieve information across multiple heterogeneous data sources. However, this is error-prone and time-consuming work since the queries can involve a sequence of multiple sub-queries over several databases. We present an implementation of an ontology-based Swine Gut Microbiota Federated Query Platform (SGMFQP) that provides a convenient, automated, and efficient query service about swine feeding and gut microbiota. The system is constructed based on a domain-specific Swine Gut Microbiota Ontology (SGMO), which facilitates the construction of queries independent of the actual organization of the data in the individual sources. This process is supported by a template-based query interface. A Datalog+-based federated query engine transforms the queries into sub-queries tailored for each individual data source, and an automated workflow orchestration mechanism executes the queries in each source database and consolidates the results. The efficiency of the system is demonstrated on several swine feeding scenarios.

X-Stream: Accelerating streaming segments on MPSoCs for real-time applications

We are witnessing a race to meet the ever-growing computation requirements of emerging AI applications to provide perception and control in autonomous vehicles — e.g., self-driving cars and UAVs. To remain competitive, vendors are packing more processing units (CPUs, programmable logic, GPUs, and hardware accelerators) into next-generation multiprocessor systems-on-a-chip (MPSoC). As a result, modern embedded platforms are achieving new heights in peak computational capacity. Unfortunately, however, the collateral and inevitable increase in complexity represents a major obstacle for the development of correct-by-design safety-critical real-time applications. Due to the ever-growing gap between fast-paced hardware evolution and comparatively slower evolution of real-time operating systems (RTOS), there is a need for real-time oriented full-platform management frameworks to complement traditional RTOS designs.In this work, we propose one such framework, namely the X-Stream framework, for the definition, synthesis, and analysis of real-time workloads targeting state-of-the-art accelerator-augmented embedded platforms. Our X-Stream framework is designed around two cardinal principles. First, computation and data movements are orchestrated to achieve predictability by design. For this purpose, iterative computation over large data chunks is divided into subsequent segments. These segments are then streamed leveraging the three-phase execution model (load, execute and unload). Second, the framework is workflow-centric: system designers can specify their workflow and the necessary code for workflow orchestration is automatically generated.In addition to automating the deployment of user-defined hardware-accelerated workloads, X-Stream supports the deployment of some computation segments on traditional CPUs. Finally, X-Stream allows the definition of real-time partitions. Each partition groups applications belonging to the same criticality level and that share the same set of hardware resources, with support for preemptive priority-driven scheduling. Conversely, freedom from interference for applications deployed in different partitions is guaranteed by design. We provide a full-system implementation that includes RTOS integration and showcase the proposed X-Stream framework on a Xilinx Ultrascale+ platform by focusing on a matrix-multiplication and addition kernel use-case.

A Machine Learning Approach for an HPC Use Case: the Jobs Queuing Time Prediction

High-Performance Computing (HPC) domain provided the necessary tools to support the scientific and industrial advancements we all have seen during the last decades. HPC is a broad domain targeting to provide both software and hardware solutions as well as envisioning methodologies that allow achieving goals of interest, such as system performance and energy efficiency. In this context, supercomputers have been the vehicle for developing and testing the most advanced technologies since their first appearance. Unlike cloud computing resources that are provided to the end-users in an on-demand fashion in the form of virtualized resources (i.e., virtual machines and containers), supercomputers’ resources are generally served through State-of-the-Art batch schedulers (e.g., SLURM, PBS, LSF, HTCondor). As such, the users submit their computational jobs to the system, which manages their execution with the support of queues. In this regard, predicting the behaviour of the jobs in the batch scheduler queues becomes worth it. Indeed, there are many cases where a deeper knowledge of the time experienced by a job in a queue (e.g., the submission of check-pointed jobs or the submission of jobs with execution dependencies) allows exploring more effective workflow orchestration policies. In this work, we focused on applying machine learning (ML) techniques to learn from the historical data collected from the queuing system of real supercomputers, aiming at predicting the time spent on a queue by a given job. Specifically, we applied both unsupervised learning (UL) and supervised learning (SL) techniques to define the most effective features for the prediction task and the actual prediction of the queue waiting time. For this purpose, two approaches have been explored: on one side, the prediction of ranges on jobs’ queuing times (classification approach) and, on the other side, the prediction of the waiting time at the minutes level (regression approach). Experimental results highlight the strong relationship between the SL models’ performances and the way the dataset is split. At the end of the prediction step, we present the uncertainty quantification approach, i.e., a tool to associate the predictions with reliability metrics, based on variance estimation.

Deep learning approach to security enforcement in cloud workflow orchestration

Supporting security and data privacy in cloud workflows has attracted significant research attention. For example, private patients’ data managed by a workflow deployed on the cloud need to be protected, and communication of such data across multiple stakeholders should also be secured. In general, security threats in cloud environments have been studied extensively. Such threats include data breaches, data loss, denial of service, service rejection, and malicious insiders generated from issues such as multi-tenancy, loss of control over data and trust. Supporting the security of a cloud workflow deployed and executed over a dynamic environment, across different platforms, involving different stakeholders, and dynamic data is a difficult task and is the sole responsibility of cloud providers. Therefore, in this paper, we propose an architecture and a formal model for security enforcement in cloud workflow orchestration. The proposed architecture emphasizes monitoring cloud resources, workflow tasks, and the data to detect and predict anomalies in cloud workflow orchestration using a multi-modal approach that combines deep learning, one class classification, and clustering. It also features an adaptation scheme to cope with anomalies and mitigate their effect on the workflow cloud performance. Our prediction model captures unsupervised static and dynamic features as well as reduces the data dimensionality, which leads to better characterization of various cloud workflow tasks, and thus provides better prediction of potential attacks. We conduct a set of experiments to evaluate the proposed anomaly detection, prediction, and adaptation schemes using a real COVID-19 dataset of patient health records. The results of the training and prediction experiments show high anomaly prediction accuracy in terms of precision, recall, and F1 scores. Other experimental results maintained a high execution performance of the cloud workflow after applying adaptation strategy to respond to some detected anomalies. The experiments demonstrate how the proposed architecture prevents unnecessary wastage of resources due to anomaly detection and prediction.