Workflow Language Research Articles

Simplifying the development of portable, scalable, and reproducible workflows.

Command-line software plays a critical role in biology research. However, processes for installing and executing software differ widely. The Common Workflow Language (CWL) is a community standard that addresses this problem. Using CWL, tool developers can formally describe a tool's inputs, outputs, and other execution details. CWL documents can include instructions for executing tools inside software containers. Accordingly, CWL tools are portable-they can be executed on diverse computers-including personal workstations, high-performance clusters, or the cloud. CWL also supports workflows, which describe dependencies among tools and using outputs from one tool as inputs to others. To date, CWL has been used primarily for batch processing of large datasets, especially in genomics. But it can also be used for analytical steps of a study. This article explains key concepts about CWL and software containers and provides examples for using CWL in biology research. CWL documents are text-based, so they can be created manually, without computer programming. However, ensuring that these documents conform to the CWL specification may prevent some users from adopting it. To address this gap, we created ToolJig, a Web application that enables researchers to create CWL documents interactively. ToolJig validates information provided by the user to ensure it is complete and valid. After creating a CWL tool or workflow, the user can create 'input-object' files, which store values for a particular invocation of a tool or workflow. In addition, ToolJig provides examples of how to execute the tool or workflow via a workflow engine. ToolJig and our examples are available at https://github.com/srp33/ToolJig.

Making Canonical Workflow Building Blocks interoperable across workflow languages

Making Canonical Workflow Building Blocks interoperable across workflow languages

A Data Centre Configurable Data Mining Document Management Information System

Data extraction is often a dynamic process that can be easily modelled as a workflow for data processing. When massive collections of data have to be evaluated and/or sophisticated data mining algorithms have to be performed, it can take very long to execute data analysis workflows. Effective technologies are also needed to incorporate flexible data collection workflows through the use of cloud-based storage platforms, where data is stored even more regularly. The paper attempts to show how cloud infrastructure is implemented to introduce an optimised framework in which scalable data analyzation workflows can be planned and performed. We explain how the Data Mining Cloud Architecture is built and applied and a data analytics method that incorporates visual workflow vocabulary, parallel to the Virtualized environment. DMCF is developed with a view to simplifying the creation of applications for data mining associated with generic system monitoring schemes that are not created especially for this area, in view of the specifications of actual data mining applications. The effects are a high-level environment that minimises the programming effort with an optimised visual workflow language, allowing the implementation of typical patterns meant to generate and execute data mining application in parallel simple to professional developers. The wall mounted of the workflow, device design and mechanisms of the DMCF are shown. We also address many DMCF-developed data mining business processes and the scalability achieved by running business processes in a cloud environment.

The Dockstore: enhancing a community platform for sharing reproducible and accessible computational protocols.

Dockstore (https://dockstore.org/) is an open source platform for publishing, sharing, and finding bioinformatics tools and workflows. The platform has facilitated large-scale biomedical research collaborations by using cloud technologies to increase the Findability, Accessibility, Interoperability and Reusability (FAIR) of computational resources, thereby promoting the reproducibility of complex bioinformatics analyses. Dockstore supports a variety of source repositories, analysis frameworks, and language technologies to provide a seamless publishing platform for authors to create a centralized catalogue of scientific software. The ready-to-use packaging of hundreds of tools and workflows, combined with the implementation of interoperability standards, enables users to launch analyses across multiple environments. Dockstore is widely used, more than twenty-five high-profile organizations share analysis collections through the platform in a variety of workflow languages, including the Broad Institute's GATK best practice and COVID-19 workflows (WDL), nf-core workflows (Nextflow), the Intergalactic Workflow Commission tools (Galaxy), and workflows from Seven Bridges (CWL) to highlight just a few. Here we describe the improvements made over the last four years, including the expansion of system integrations supporting authors, the addition of collaboration features and analysis platform integrations supporting users, and other enhancements that improve the overall scientific reproducibility of Dockstore content.

A Review on Design and Development of Performance Evaluation Model for Bio-Informatics Data Using Hadoop

The paper reviews the usage of the platform Hadoop in applications for systemic bioinformatics. Hadoop offers another system for Structural Bioinformatics to break down broad fractions of the Protein Data Bank that is crucial to high-throughput investigations of (for example) protein-ligand docking, protein-ligand complex clustering, and structural alignment. In specific, we review different applications of high-throughput analyses and their scalability in the literature using Hadoop. In comparison to revising the algorithms, we find that these organisations typically use a realized executable called MapReduce. Scalability demonstrates variable behavior in correlation with other batch schedulers, particularly as immediate examinations are usually not accessible on a similar platform. Direct Hadoop examinations with batch schedulers are missing in the literature, but we note that there is some evidence that the scale of MPI executions is better than Hadoop. The dilemma of the interface and structure of an asset to use Hadoop is a significant obstacle to the utilization of the Hadoop biological framework. This will enhance additional time as Hadoop interfaces, such as enhancing Flash, increasing the use of cloud platforms, and normalized approaches, for example, are taken up by Workflow Languages.

Bioconductor toolchain for reproducible bioinformatics pipelines using Rcwl and RcwlPipelines.

SummaryThe Common Workflow Language (CWL) is used to provide portable and reproducible data analysis workflows across different tools and computing environments. We have developed Rcwl, an R interface to CWL, to provide easier development, use and maintenance of CWL pipelines from within R. We have also collected more than 100 pre-built tools and pipelines in RcwlPipelines, ready to be queried and used by researchers in their own analysis. A single-cell RNA sequencing preprocessing pipeline demonstrates use of the software.Availability and implementationProject website: https://rcwl.org (Rcwl: https://bioconductor.org/packages/Rcwl; RcwlPipelines: https://bioconductor.org/packages/RcwlPipelines).Supplementary information Supplementary data are available at Bioinformatics online.

Making Canonical Workflow Building Blocks interoperable across workflow languages

Making Canonical Workflow Building Blocks interoperable across workflow languages

A workflow language for research e-infrastructures

Research e-infrastructures are “systems of systems,” patchworks of resources such as tools and services, which change over time to address the evolving needs of the scientific process. In such environments, researchers carry out their scientific process in terms of sequences of actions that mainly include invocation of web services, user interaction with web applications, user download and use of shared software libraries/tools. The resulting workflows are intended to generate new research products (articles, datasets, methods, etc.) out of existing ones. Sharing a digital and executable representation of such workflows with other scientists would enforce Open Science publishing principles of “reproducibility of science” and “transparent assessment of science.” This work presents HyWare, a language and execution platform capable of representing scientific processes in highly heterogeneous research e-infrastructures in terms of so-called hybrid workflows. Hybrid workflows can express sequences of “manually executable actions,” i.e., formal descriptions guiding users to repeat a reasoning, protocol or manual procedure, and “machine-executable actions,” i.e., encoding of the automated execution of one (or more) web services. An HyWare execution platform enables scientists to (i) create and share workflows out of a given action set (as defined by the users to match e-infrastructure needs) and (ii) execute hybrid workflows making sure input/output of the actions flow properly across manual and automated actions. The HyWare language and platform can be implemented as an extension of well-known workflow languages and platforms.

PM4NGS, a project management framework for next-generation sequencing data analysis.

FAIR (Findability, Accessibility, Interoperability, and Reusability) next-generation sequencing (NGS) data analysis relies on complex computational biology workflows and pipelines to guarantee reproducibility, portability, and scalability. Moreover, workflow languages, managers, and container technologies have helped address the problem of data analysis pipeline execution across multiple platforms in scalable ways. Here, we present a project management framework for NGS data analysis called PM4NGS. This framework is composed of an automatic creation of a standard organizational structure of directories and files, bioinformatics tool management using Docker or Bioconda, and data analysis pipelines in CWL format. Pre-configured Jupyter notebooks with minimum Python code are included in PM4NGS to produce a project report and publication-ready figures. We present 3 pipelines for demonstration purposes including the analysis of RNA-Seq, ChIP-Seq, and ChIP-exo datasets. PM4NGS is an open source framework that creates a standard organizational structure for NGS data analysis projects. PM4NGS is easy to install, configure, and use by non-bioinformaticians on personal computers and laptops. It permits execution of the NGS data analysis on Windows 10 with the Windows Subsystem for Linux feature activated. The framework aims to reduce the gap between researcher in experimental laboratories producing NGS data and workflows for data analysis. PM4NGS documentation can be accessed at https://pm4ngs.readthedocs.io/.

Software-Defined Workflows for Distributed Interoperable Closed-Loop Neuromodulation Control Systems

Closed-loop neuromodulation control systems facilitate regulating abnormal physiological processes by recording neurophysiological activities and modifying those activities through feedback loops. Designing such systems requires interoperable service composition, consisting of cycles. Workflow frameworks enable standard modular architectures, offering reproducible automated pipelines. However, those frameworks limit their support to executions represented by directed acyclic graphs (DAGs). DAGs need a pre-defined start and end execution step with no cycles, thus preventing the researchers from using the standard workflow languages as-is for closed-loop workflows and pipelines. In this paper, we present NEXUS, a workflow orchestration framework for distributed analytics systems. NEXUS proposes a Software-Defined Workflows approach, inspired by Software-Defined Networking (SDN), which separates the data flows across the service instances from the control flows. NEXUS enables creating interoperable workflows with closed loops by defining the workflows in a logically centralized approach, from microservices representing each execution step. The centralized NEXUS orchestrator facilitates dynamically composing and managing scientific workflows from the services and existing workflows, with minimal restrictions. NEXUS represents complex workflows as directed hypergraphs (DHGs) rather than DAGs. We illustrate a seamless execution of neuromodulation control systems by supporting loops in a workflow as the use case of NEXUS. Our evaluations highlight the feasibility, flexibility, performance, and scalability of NEXUS in modeling and executing closed-loop workflows.

ACLImatise: automated generation of tool definitions for bioinformatics workflows.

SummaryaCLImatise is a utility for automatically generating tool definitions compatible with bioinformatics workflow languages, by parsing command-line help output. aCLImatise also has an associated database called the aCLImatise Base Camp, which provides thousands of pre-computed tool definitions.Availability and implementationThe latest aCLImatise source code is available within a GitHub organisation, under the GPL-3.0 license: https://github.com/aCLImatise. In particular, documentation for the aCLImatise Python package is available at https://aclimatise.github.io/CliHelpParser/, and the aCLImatise Base Camp is available at https://aclimatise.github.io/BaseCamp/.Supplementary information Supplementary data are available at Bioinformatics online.

A contract-based workflow execution framework for realizing artifact-centric business processes in a dynamic and collaborative environment

PurposeThe paper aims to study realization requirements for the flexible enactment of artifact-centric business processes in a dynamic, collaborative environment and to develop a workflow execution framework that can effectively address those requirements.Design/methodology/approachThis study proposed a framework and contract-based, event-driven architecture design and implementation that can directly realize collaborative artifact-centric business processes in service-oriented architecture (SOA) without any model conversion.FindingsThe results show that the approach is feasible in presenting several key benefits over the use of existing workflow systems to run artifact-centric processes.Originality/valueMost of the existing approaches require an artifact-centric model to be transformed into executable workflow languages to run on existing workflow management systems. This study argues that the model conversion can incur losses of information and affect traceability and monitoring ability of workflows, especially in an SOA where a workflow can span across multiple inter-business entities.

VIRTUS: a pipeline for comprehensive virus analysis from conventional RNA-seq data.

SummaryThe possibility that RNA transcripts from clinical samples contain plenty of virus RNAs has not been pursued actively so far. We here developed a new tool for analyzing virus-transcribed mRNAs, not virus copy numbers, in the data of bulk and single-cell RNA-sequencing of human cells. Our pipeline, named VIRTUS (VIRal Transcript Usage Sensor), was able to detect 762 viruses including herpesviruses, retroviruses and even SARS-CoV-2 (COVID-19), and quantify their transcripts in the sequence data. This tool thus enabled simultaneously detecting infected cells, the composition of multiple viruses within the cell, and the endogenous host-gene expression profile of the cell. This bioinformatics method would be instrumental in addressing the possible effects of covertly infecting viruses on certain diseases and developing new treatments to target such viruses.Availability and implementation: VIRTUS is implemented using Common Workflow Language and Docker under a CC-NC license. VIRTUS is freely available at https://github.com/yyoshiaki/VIRTUS.Supplementary information Supplementary data are available at Bioinformatics online.

BCO App: tools for generating BioCompute Objects from next-generation sequencing workflows and computations

The BioCompute Object (BCO) standard is an IEEE standard (IEEE 2791-2020) designed to facilitate the communication of next-generation sequencing data analysis with applications across academia, government agencies, and industry. For example, the Food and Drug Administration (FDA) supports the standard for regulatory submissions and includes the standard in their Data Standards Catalog for the submission of HTS data. We created the BCO App to facilitate BCO generation in a range of computational environments and, in part, to participate in the Advanced Track of the precisionFDA BioCompute Object App-a-thon. The application facilitates the generation of BCOs from both workflow metadata provided as plaintext and from workflow contents written in the Common Workflow Language. The application can also access and ingest task execution results from the Cancer Genomics Cloud (CGC), an NCI funded computational platform. Creating a BCO from a CGC task significantly reduces the time required to generate a BCO on the CGC by auto-populating workflow information fields from CGC workflow and task execution results. The BCO App supports exporting BCOs as JSON or PDF files and publishing BCOs to both the CGC platform and to GitHub repositories.

BCO App: tools for generating BioCompute Objects from next-generation sequencing workflows and computations

The BioCompute Object (BCO) standard is an IEEE standard (IEEE 2791-2020) designed to facilitate the communication of next-generation sequencing data analysis with applications across academia, government agencies, and industry. For example, the Food and Drug Administration (FDA) supports the standard for regulatory submissions and includes the standard in their Data Standards Catalog for the submission of HTS data. We created the BCO App to facilitate BCO generation in a range of computational environments and, in part, to participate in the Advanced Track of the precisionFDA BioCompute Object App-a-thon. The application facilitates the generation of BCOs from both workflow metadata provided as plaintext and from workflow contents written in the Common Workflow Language. The application can also access and ingest task execution results from the Cancer Genomics Cloud (CGC), an NCI funded computational platform. Creating a BCO from a CGC task significantly reduces the time required to generate a BCO on the CGC by auto-populating workflow information fields from CGC workflow and task execution results. The BCO App supports exporting BCOs as JSON or PDF files and publishing BCOs to both the CGC platform and to GitHub repositories.

A Complete Language for Faceted Dataflow Programs

We present a complete categorical axiomatization of a wide class of dataflow programs. This gives a three-dimensional diagrammatic language for workflows, more expressive than the directed acyclic graphs generally used for this purpose. This calls for an implementation of these representations in data transformation tools.

AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance

The ever-growing availability of computing power and the sustained development of advanced computational methods have contributed much to recent scientific progress. These developments present new challenges driven by the sheer amount of calculations and data to manage. Next-generation exascale supercomputers will harden these challenges, such that automated and scalable solutions become crucial. In recent years, we have been developing AiiDA (aiida.net), a robust open-source high-throughput infrastructure addressing the challenges arising from the needs of automated workflow management and data provenance recording. Here, we introduce developments and capabilities required to reach sustained performance, with AiiDA supporting throughputs of tens of thousands processes/hour, while automatically preserving and storing the full data provenance in a relational database making it queryable and traversable, thus enabling high-performance data analytics. AiiDA’s workflow language provides advanced automation, error handling features and a flexible plugin model to allow interfacing with external simulation software. The associated plugin registry enables seamless sharing of extensions, empowering a vibrant user community dedicated to making simulations more robust, user-friendly and reproducible.

Sarek: A portable workflow for whole-genome sequencing analysis of germline and somatic variants.

Whole-genome sequencing (WGS) is a fundamental technology for research to advance precision medicine, but the limited availability of portable and user-friendly workflows for WGS analyses poses a major challenge for many research groups and hampers scientific progress. Here we present Sarek, an open-source workflow to detect germline variants and somatic mutations based on sequencing data from WGS, whole-exome sequencing (WES), or gene panels. Sarek features (i) easy installation, (ii) robust portability across different computer environments, (iii) comprehensive documentation, (iv) transparent and easy-to-read code, and (v) extensive quality metrics reporting. Sarek is implemented in the Nextflow workflow language and supports both Docker and Singularity containers as well as Conda environments, making it ideal for easy deployment on any POSIX-compatible computers and cloud compute environments. Sarek follows the GATK best-practice recommendations for read alignment and pre-processing, and includes a wide range of software for the identification and annotation of germline and somatic single-nucleotide variants, insertion and deletion variants, structural variants, tumour sample purity, and variations in ploidy and copy number. Sarek offers easy, efficient, and reproducible WGS analyses, and can readily be used both as a production workflow at sequencing facilities and as a powerful stand-alone tool for individual research groups. The Sarek source code, documentation and installation instructions are freely available at https://github.com/nf-core/sarek and at https://nf-co.re/sarek/.

Abstract 4414: Single-cell analysis on the Cancer Genomics Cloud reveals changes in the transcriptome profiles of endothelial tumor cells over time - novel insights from a public dataset of a mouse model of melanoma

Abstract The Cancer Genomics Cloud (CGC) powered by Seven Bridges makes the analysis of large datasets accessible from any environment, which is critical as the volume of datasets continues to expand. This expansion of the size, complexity, and location of datasets has been prominent in the field of single-cell genomics, which can present challenges for data analysis. Here we combined the rich datasets available through the Human Cell Atlas (HCA) with the tools and computational resources available on the CGC to provide a path for any scientist to explore complex biological questions. To demonstrate these capabilities, we processed a single-cell dataset of immune and stromal cells isolated from the tumor and lymph nodes of a mouse model of melanoma using an optimized version of the SmartSeq2 pipeline recommended by the HCA. This dataset contains 13.3k paired-end sequencing files for 6.6k estimated cells, with a total size of 371.60 GB. The entire dataset was transferred from Human Cell Atlas Data Portal to the CGC, where it is accessible to all users. The goal of this study was to process a single-cell dataset on the tumor microenvironment with the SmartSeq2 pipeline and to investigate changes in transcriptome profiles of endothelial tumor cells over time. We used an alternative approach than authors in the original study, which aimed to analyze the immune and stromal cells (Davidson et al. bioRxiv 2018). To perform alignment and quantification of raw sequencing reads using the CGC execution environment, we created an implementation of the SmartSeq2 pipeline in Common Workflow Language. Quantification results were used to perform cell clustering, followed by a pseudotime analysis on the endothelial tumor cells. For the cell clustering and pseudotime analyses, we created interactive notebooks in the RMarkdown format that can be executed within the RStudio environment on the CGC. In summary, this work demonstrates the importance of developing reproducible workflows that can run on multiple environments as well as data sharing that can bring to light new insights from the existing data. Citation Format: Nemanja Vucic, Manisha Ray, Dalibor Veljkovic, Stefan Cidilko, Brandi Davis-Dusenbery. Single-cell analysis on the Cancer Genomics Cloud reveals changes in the transcriptome profiles of endothelial tumor cells over time - novel insights from a public dataset of a mouse model of melanoma [abstract]. In: Proceedings of the Annual Meeting of the American Association for Cancer Research 2020; 2020 Apr 27-28 and Jun 22-24. Philadelphia (PA): AACR; Cancer Res 2020;80(16 Suppl):Abstract nr 4414.

Sarek: A portable workflow for whole-genome sequencing analysis of germline and somatic variants

Whole-genome sequencing (WGS) is a fundamental technology for research to advance precision medicine, but the limited availability of portable and user-friendly workflows for WGS analyses poses a major challenge for many research groups and hampers scientific progress. Here we present Sarek, an open-source workflow to detect germline variants and somatic mutations based on sequencing data from WGS, whole-exome sequencing (WES), or gene panels. Sarek features (i) easy installation, (ii) robust portability across different computer environments, (iii) comprehensive documentation, (iv) transparent and easy-to-read code, and (v) extensive quality metrics reporting. Sarek is implemented in the Nextflow workflow language and supports both Docker and Singularity containers as well as Conda environments, making it ideal for easy deployment on any POSIX-compatible computers and cloud compute environments. Sarek follows the GATK best-practice recommendations for read alignment and pre-processing, and includes a wide range of software for the identification and annotation of germline and somatic single-nucleotide variants, insertion and deletion variants, structural variants, tumour sample purity, and variations in ploidy and copy number. Sarek offers easy, efficient, and reproducible WGS analyses, and can readily be used both as a production workflow at sequencing facilities and as a powerful stand-alone tool for individual research groups. The Sarek source code, documentation and installation instructions are freely available at https://github.com/nf-core/sarek and at https://nf-co.re/sarek/.