Multiple Workflows Research Articles

Reinforcement Learning-Based Online Scheduling of Multiple Workflows in Edge Environment

Reinforcement Learning-Based Online Scheduling of Multiple Workflows in Edge Environment

Clustering-based multi-objective optimization considering fairness for multi-workflow scheduling on clouds

Distributed computing, such as cloud computing, provides promising platforms for orchestrating scientific workflows' tasks based on their sequences and dependencies. Workflow scheduling plays an important role in optimizing concerned objectives for distributed computing, such as minimizing the makespan and cost. Many researchers have focused on optimizing a specific single workflow with multiple objectives. Currently, there are few studies on multi-workflow scheduling, with most research focusing on objectives such as cost and makespan. However, multi-workflow scheduling requires the design of specific objectives that reflect the unique characteristics of multiple workflows. On the other hand, clustering-based approaches have garnered significant attention in the field of workflow scheduling over distributed computing resources due to their advantage in reducing data communication among tasks. Despite this, the effectiveness of clustering-based algorithms has not been extensively studied and validated in the context of multi-objective multi-workflow scheduling models. Motivated by these factors, we propose an approach for multiple workflows' multi-objective optimization (MOO), considering the new defined metric, fairness. We first mathematically formulate the fairness and define a fairness-involved MOO model. Then, we propose an advanced clustering-based resource optimization strategy in multiple workflow runs. Experimental results show that the proposed approach performs better than the compared algorithms without significant compromise of the overall makespan and cost as well as individual fairness, which can guide the simulation workflow scheduling on clouds.

Interactome of the HIV-1 proteome and human host RNA

The human immunodeficiency virus (HIV-1) is highly dependent on a variety of host factors. Beside proteins, host RNA molecules are reported to aid HIV-1 replication and latency maintenance. Here, we implement multiple workflows of native RNA immunoprecipitation and sequencing (nRIPseq) to determine direct host RNA interaction partners of all 18 HIV-1 (poly)proteins. We identify 1,727 HIV-1 protein – human RNA interactions in the Jurkat cell line and 1,558 interactions in SupT1 cells for a subset of proteins, and discover distinct cellular pathways that seem to be used or controlled by HIV-1 on the RNA level: Tat binds mRNAs of proteins involved in the super elongation complex (AFF1-4, Cyclin-T1). Correlation of the interaction scores (based on binding abundancy) allows identifying the highest confidence interactions, for which we perform a small-scale knockdown screen that leads to the identification of three HIV-1 protein binding RNA interactors involved in HIV-1 replication (AFF2, H4C9 and RPLP0).

CMK: Enhancing Resource Usage Monitoring across Diverse Bioinformatics Workflow Management Systems

The increasing use of multiple Workflow Management Systems (WMS) employing various workflow languages and shared workflow repositories enhances the open-source bioinformatics ecosystem. Efficient resource utilization in these systems is crucial for keeping costs low and improving processing times, especially for large-scale bioinformatics workflows running in cloud environments. Recognizing this, our study introduces a novel reference architecture, Cloud Monitoring Kit (CMK), for a multi-platform monitoring system. Our solution is designed to generate uniform, aggregated metrics from containerized workflow tasks scheduled by different WMS. Central to the proposed solution is the use of task labeling methods, which enable convenient grouping and aggregating of metrics independent of the WMS employed. This approach builds upon existing technology, providing additional benefits of modularity and capacity to seamlessly integrate with other data processing or collection systems. We have developed and released an open-source implementation of our system, which we evaluated on Amazon Web Services (AWS) using a transcriptomics data analysis workflow executed on two scientific WMS. The findings of this study indicate that CMK provides valuable insights into resource utilization. In doing so, it paves the way for more efficient management of resources in containerized scientific workflows running in public cloud environments, and it provides a foundation for optimizing task configurations, reducing costs, and enhancing scheduling decisions. Overall, our solution addresses the immediate needs of bioinformatics workflows and offers a scalable and adaptable framework for future advancements in cloud-based scientific computing.

Sapporo: A workflow execution service that encourages the reuse of workflows in various languages in bioinformatics.

The increased demand for efficient computation in data analysis encourages researchers in biomedical science to use workflow systems. Workflow systems, or so-called workflow languages, are used for the description and execution of a set of data analysis steps. Workflow systems increase the productivity of researchers, specifically in fields that use high-throughput DNA sequencing applications, where scalable computation is required. As systems have improved the portability of data analysis workflows, research communities are able to share workflows to reduce the cost of building ordinary analysis procedures. However, having multiple workflow systems in a research field has resulted in the distribution of efforts across different workflow system communities. As each workflow system has its unique characteristics, it is not feasible to learn every single system in order to use publicly shared workflows. Thus, we developed Sapporo, an application to provide a unified layer of workflow execution upon the differences of various workflow systems. Sapporo has two components: an application programming interface (API) that receives the request of a workflow run and a browser-based client for the API. The API follows the Workflow Execution Service API standard proposed by the Global Alliance for Genomics and Health. The current implementation supports the execution of workflows in four languages: Common Workflow Language, Workflow Description Language, Snakemake, and Nextflow. With its extensible and scalable design, Sapporo can support the research community in utilizing valuable resources for data analysis.

Navigating the complexity of detrital rutile provenance: methodological insights from the Neotethys Orogen in Anatolia

Abstract. Sedimentary provenance is a powerful tool for reconstructing convergent margin evolution. However, single mineral approaches, like detrital zircon, have struggled to track sediment input from mafic and metamorphic sources. Detrital rutile complements detrital zircon datasets by offering a path forward in sedimentary provenance reconstructions where metamorphic terranes are potential source regions. However, U–Pb geochronology in rutile can be difficult due to low uranium concentrations and incorporation of common Pb, and multiple workflows are currently in use. Here, we investigate U–Pb and trace element data reduction, processing, and common Pb correction workflows using new detrital rutile U–Pb geochronology and trace element geochemistry results from the Late Cretaceous to Eocene Central Sakarya and Sarıcakaya basins in Anatolia. A significant number of analyses were rejected (54 %) due to signal intensity limitations, namely low U, low Pb, anomalous signal, and inclusions. We identify this as a universal limitation of large-n detrital rutile studies and recommend the systematic reporting of the amount of discarded analysis and the processes for rejection in all studies using detrital rutile U–Pb geochronology. Additionally, we show that (1) the 208Pb and 207Pb common Pb reduction schemes produce similar age distributions and can be used interchangeably, while (2) the Stacey–Kramers distance is a suitable metric for quantifying U–Pb discordance, but a discordance filter is not recommended. (3) Instead, filtering U–Pb data by a power law function based on the corrected date uncertainty is appropriate. (4) The exclusion of low uranium concentration rutile biases date distributions and favors pelitic-derived, higher Zr-in-rutile temperature, and higher U–Pb concordance grains. (5) Paired U–Pb and trace elements can be used to evaluate potential bias in U–Pb data rejection, which reveals that data rejection does not bias the provenance interpretations. Finally, (6) The signature of sediment recycling can be identified through U–Pb dates and Zr-in-rutile temperatures. To better navigate the complexity of detrital rutile datasets and to facilitate the standardization of data reporting approaches, we provide open-access code as Jupyter notebooks for data processing and analysis steps, including common Pb corrections, uncertainty filters, discordance calculations, and trace element analysis.

A Survey on Multiple Workflow Scheduling Algorithms in Cloud Environment

The workflow approach is a standard for displaying processes and their implementation process. With the advent of electrical sciences, more cumbersome workflows were created with more processing requirements. New distributed systems, such as grid systems and computing clouds, allow users to access heterogeneous resources that are geographically located at different points and execute their workflow tasks. Therefore, the simultaneous receipt and execution of several workflows is obvious. As a result of discussing scheduling algorithms, it is necessary to consider arrangements for implementing multiple workflows on a shared resource set. Improving the execution of multiple workflows can accelerate the process of obtaining results when sending processes to the cloud. In this paper, we first discuss the classification of multiple workflow scheduling algorithms, and then briefly describe the scheduling algorithms for the cloud environment, andfinally, the algorithms of papers were compared with each other

Flipping: A Valve-Free Strategy to Control Fluid Flow in Centrifugal Microfluidic Systems

Centrifugal microfluidics has emerged as a technology to automate fluid handling at the microliter scale in bioassays. It has been mostly developed for point-of-care diagnostic applications, where assays involve a well-defined workflow. By contrast, this study introduces a microfluidic strategy that allows multiple workflows within the same device, thereby addressing the requirements of sample preparation steps in R&D settings. The corresponding device is a microfluidic chip that can be inserted either in a lab-on-a-disk or in the swinging-bucket of a commercial centrifuge. The control of liquid transfers from one chamber to the next is achieved by flipping the chip, thereby reorienting it in the centrifugal force field. The present experimental study describes the fluid instability that triggers the transfer upon flipping. A semi-empirical physical model is developed that predicts the acceleration threshold of the instability as a function of design parameters. By contrast to most existing control strategies in centrifugal microfluidics, the present strategy does not involve any valve. Consequently, it is robust to variations of material and sample properties, and it yields minimal fabrication constraints. Designing appropriately sized chambers enables flexible implementation of complex workflows. This strategy brings the miniaturization benefits of centrifugal microfluidics to sample preparation workflows.

Integrated Microfluidics for Single‐Cell Separation and On‐Chip Analysis: Novel Applications and Recent Advances

From deciphering infection and disease mechanisms to identifying novel biomarkers and personalizing treatments, the characteristics of individual cells can provide significant insights into a variety of biological processes and facilitate decision‐making in biomedical environments. Conventional single‐cell analysis methods are limited in terms of cost, contamination risks, sample volumes, analysis times, throughput, sensitivity, and selectivity. Although microfluidic approaches have been suggested as a low‐cost, information‐rich, and high‐throughput alternative to conventional single‐cell isolation and analysis methods, limitations such as necessary off‐chip sample pre‐ and post‐processing as well as systems designed for individual workflows have restricted their applications. In this review, a comprehensive overview of recent advances in integrated microfluidics for single‐cell isolation and on‐chip analysis in three prominent application domains are provided: investigation of somatic cells (particularly cancer and immune cells), stem cells, and microorganisms. Also, the use of conventional cell separation methods (e.g., dielectrophoresis) in unconventional or novel ways, which can advance the integration of multiple workflows in microfluidic systems, is discussed. Finally, a critical discussion related to current limitations of integrated microfluidic single‐cell workflows and how they could be overcome is provided.

Genetic-Based Two Granularity Ordering Methods for Multiple Workflow Scheduling

Genetic-Based Two Granularity Ordering Methods for Multiple Workflow Scheduling

A Novel Optimization Approach for Energy-Efficient Multiple Workflow Scheduling in Cloud Environment

A Novel Optimization Approach for Energy-Efficient Multiple Workflow Scheduling in Cloud Environment

A metatranscriptomic evaluation of viruses in field-collected bed bugs.

Cimex lectularius, known as the common bed bug, is a widespread hematophagous human ectoparasite and urban pest that is not known to be a vector of any human infectious disease agents. However, few studies in the era of molecular biology have profiled the microorganisms harbored by field populations of bed bugs. The objective of this study was to examine the viruses present in a large sampling of common bed bugs and related bat bugs (Cimex pipistrelle). RNA sequencing was undertaken on an international sampling of > 500 field-collected bugs, and multiple workflows were used to assemble contigs and query these against reference nucleotide databases to identify viral genomes. Shuangao bed bug virus 2, an uncharacterized rhabdovirus previously discovered in Cimex hemipterus from China, was found in several bed bug pools from the USA and Europe, as well as in C. pipistrelle, suggesting that this virus is common among bed bug populations. In addition, Shuangao bed bug virus 1 was detected in a bed bug pool from China, and sequences matching Enterobacteria phage P7 were found in all bed bug pools, indicating the ubiquitous presence of phage-derived elements in the genome of the bed bug or its enterobacterial symbiont. However, viral diversity was low in bed bugs in our study, as no other viral genomes were detected with significant coverage. These results provide evidence against frequent virus infection in bed bugs. Nonetheless, our investigation had several important limitations, and additional studies should be conducted to better understand the prevalence and composition of viruses in bed bugs. Most notably, our study largely focused on insects from urban areas in industrialized nations, thus likely missing infrequent virus infections and those that could occur in rural or tropical environments or developing nations.

Co-evolutionary and Elite learning-based bi-objective Poor and Rich Optimization algorithm for scheduling multiple workflows in the cloud

Cloud computing is a cost-effective environment for deploying large-scale scientific applications. However, multi-workflow scheduling has great challenge since users may request a series of applications with different Quality of Service (QoS) at the same time. In this paper, a Co-evolutionary and Elite learning-based bi-objective Poor and Rich Optimization algorithm (CE-PRO) is proposed for scheduling applications to minimize the makespan and cost of each workflow. First, an MPMO framework is combined with PRO to optimize two objectives by two populations, respectively for better balancing the search diversity and convergence speed, where each population is updated by an improved PRO, which adopts the middle-class sub-population and re-defines the update mechanism for rich individuals to enhance search diversity and reduce the possibility of falling into local optima. Second, to restrain each population focusing overly on its respective objective, a global information exchange pool is innovatively designed to save the non-dominated solutions ever found, which will be used back as the shared guiding solutions to foster inter-population communication and co-evolution during an evolutionary process. Third, a hybrid mutation-based Elite Enhancement Strategy (EES) is developed by introducing multiple scales of mutation operations into elite solutions alternatively and iteratively to exploit excellent individuals and explore more trade-off solutions. Extensive experiments are conducted on real world scientific workflows with different types and scales, and the experimental results demonstrate that in most cases, our proposed CE-PRO outperforms its peers in the number of obtained non-dominated solutions, and the solution diversity and quality as well. In particular, the dominance of CE-PRO is superior to its peers by at least 25.62%.

Dual-mutation mechanism-driven snake optimizer for scheduling multiple budget constrained workflows in the cloud

Scheduling workflows in cloud computing is to find the most appropriate mapping from a series of dependent tasks to a set of available virtual resources so as to minimize or maximize some user concerned objectives, which is very significant to a sustainable and high-efficient cloud data center. However, it still faces challenges since its NP-hardness and the diversified requirements of both the cloud service consumers and cloud service providers need to be satisfied at the same time, especially in addressing multiple applications requested simultaneously. This work proposes a dual-mutation mechanism-driven snake optimizer for scheduling multiple workflows in the cloud for minimizing the makespan of each workflow under user pre-defined budget constraints. Firstly, a continuous optimization algorithm, namely Snake Optimizer (SO) is adopted into discrete optimization, i.e., workflow scheduling. Secondly, a task execution order aware fitness function is designed to reduce the gaps or waiting time between parent and child tasks within a workflow and thus reduce the total execution time of the workflow. Besides, we analyze the existing snake optimizer and adjust the parameters corresponding to different evolutionary stages to adapt to our considered problem. Finally, a dual-mutation mechanism is developed by introducing a non-improvement iteration number for each snake and applying a standard bit mutation operation to prematurely converging snakes and the individuals randomly selected from the remaining snakes so that the population diversity can be enhanced and more potential solutions can be explored. Experimental results on a set of real-world scientific workflows show that our proposed algorithm is of great superiority in constraint satisfiability, meaning that compared with its peers, it is always the first to find feasible solutions.

Mapping faults in 3D seismic data – why the method matters

3D seismic reflection imagery is the most widely used resource for interpreting the geometric structure of faults in the subsurface. Yet these endeavours carry uncertainties, the significance of which are rarely discussed. We explore how the application of different workflows yield different interpretations of a single high-quality 3D seismic image-set. We describe and apply five mapping workflows, based on 2D derivations of imagery to map an array of small-scale faults. Some workflows use vertical profiles, a conventional approach, others use plan views. We also vary the amount of under-sampling. Stacking the fault maps derived from the five workflows into a heat map shows broadly similar trends and distributions of faults irrespective of the workflow deployed. However, juxtaposition mapping reveals differences in fault length and network pattern (linkage and segmentation) arising from the different workflows. We quantify the total fault areas and distribution of fault lengths for each workflow – revealing significant differences in these statistics. Our results show that mapping strategies impact the interpretation of fault geometry, their network patterns and derived statistics. This understanding is critical for assessing and risking fault interpretations – deploying multiple workflows can reveal inherent uncertainty in structural interpretation of 3D seismic imagery.

Sharing the Joy of Music: A Music Cataloger’s Guide to Training Library Staff with Limited Music Expertise

ABSTRACT Countless academic libraries are unable to commit staff time to handle the multiple workflows required by a music library, such as ordering, procurement, and cataloging. Music librarians often rely on support from staff in other departments with limited musical backgrounds. Drawing from resources offered by the central library, a University of Tennessee music cataloger created guidelines for training non-musical library staff in identifying various musical formats. The training was meant to improve the workflow of music materials from acquisitions, cataloging, and processing departments. The training focused on illustrating the differences between a variety of commonly found musical score formats. The goal was to increase staff effectiveness and reduce errors in ordering and processing music materials. Ultimately, the training enabled library staff to identify best options and apply sound judgment when fulfilling order requests. These guidelines included illustrations with examples of different formats with cover images, title pages, music notation pages, and a brief bibliographic description of MARC fields 100, 245, 240, 250 and 300. By evaluating bibliographic records from WorldCat and showing physical examples of various musical manifestations, non-musical library staff were able to contribute more effectively to the operations of the music library. Furthermore, during the process of training and following up with staff, it became clear that the need for basic music training was greater than initially perceived and that numerous other units within the library needed this training. This paper offers guidelines for music library training from both the University of Tennessee and the music library literature.

An Energy-Efficient Dynamic Scheduling Method of Deadline-Constrained Workflows in a Cloud Environment

With the rapid development of cloud applications, the computing requests of cloud data centers have increased significantly, consuming a lot of energy, making cloud data centers unsustainable, which is very unfavorable from both the cloud provider’s point of view and the environmental point of view. Therefore, it is crucial to minimize energy consumption and improve resource utilization while ensuring user service quality constraints. In this paper, we propose a hybrid workflow scheduling algorithm (Online Hybrid Dynamic Scheduling, OHDS), which aims to minimize the energy consumption of tasks and maximize service resource utilization while satisfying the sub-deadline and data dependency constraints of workflow tasks. Firstly, the data dependencies between workflow tasks are considered for multi-task merging, and sub-deadline constraints are assigned to workflow tasks based on task priority. Secondly, based on the independent nature of the tasks of different workflows, a hybrid scheduling of multiple workflows is performed to reduce service idle time. Then, the workflow task scheduling priority and its sub-deadlines are dynamically adjusted, and the service status is sensed by the CPU utilization of the service, and the workload on the overloaded/underloaded service is balanced by dynamic migration of virtual machines. Finally, the OHDS method is compared with three existing scheduling methods to verify its better performance in terms of scheduling energy consumption, scheduling success rate and service resource utilization.

A narrative review of recent tools and innovations toward automating living systematic reviews and evidence syntheses

Living reviews are an increasingly popular research paradigm. The purpose of a ‘living’ approach is to allow rapid collation, appraisal and synthesis of evolving evidence on an important research topic, enabling timely influence on patient care and public health policy. However, living reviews are time- and resource-intensive. The accumulation of new evidence and the possibility of developments within the review’s research topic can introduce unique challenges into the living review workflow.To investigate the potential of software tools to support living systematic or rapid reviews, we present a narrative review informed by an examination of tools contained on the Systematic Review Toolbox website. We identified 11 tools with relevant functionalities and discuss the important features of these tools with respect to different steps of the living review workflow. Four tools (NestedKnowledge, SWIFT-ActiveScreener, DistillerSR, EPPI-Reviewer) covered multiple, successive steps of the review process, and the remaining tools addressed specific components of the workflow, including scoping and protocol formulation, reference retrieval, automated data extraction, write-up and dissemination of data.We identify several ways in which living reviews can be made more efficient and practical. Most of these focus on general workflow management, or automation through artificial intelligence and machine-learning, in the screening process. More sophisticated uses of automation mostly target living rapid reviews to increase the speed of production or evidence maps to broaden the scope of the map. We use a case study to highlight some of the barriers and challenges to incorporating tools into the living review workflow and processes. These include increased workload, the need for organisation, ensuring timely dissemination and challenges related to the development of bespoke automation tools to facilitate the review process. We describe how current end-user tools address these challenges, and which knowledge gaps remain that could be addressed by future tool development.Dedicated web presences for automatic dissemination of in-progress evidence updates, rather than solely relying on peer-reviewed journal publications, help to make the effort of a living evidence synthesis worthwhile. Despite offering basic living review functionalities, existing end-user tools could be further developed to be interoperable with other tools to support multiple workflow steps seamlessly, to address broader automatic evidence retrieval from a larger variety of sources, and to improve dissemination of evidence between review updates.

CryoGrid-PIXUL-RNA: high throughput RNA isolation platform for tissue transcript analysis

BackgroundDisease molecular complexity requires high throughput workflows to map disease pathways through analysis of vast tissue repositories. Great progress has been made in tissue multiomics analytical technologies. To match the high throughput of these advanced analytical platforms, we have previously developed a multipurpose 96-well microplate sonicator, PIXUL, that can be used in multiple workflows to extract analytes from cultured cells and tissue fragments for various downstream molecular assays. And yet, the sample preparation devices, such as PIXUL, along with the downstream multiomics analytical capabilities have not been fully exploited to interrogate tissues because storing and sampling of such biospecimens remain, in comparison, inefficient.ResultsTo mitigate this tissue interrogation bottleneck, we have developed a low-cost user-friendly system, CryoGrid, to catalog, cryostore and sample tissue fragments. TRIzol is widely used to isolate RNA but it is labor-intensive, hazardous, requires fume-hoods, and is an expensive reagent. Columns are also commonly used to extract RNA but they involve many steps, are prone to human errors, and are also expensive. Both TRIzol and column protocols use test tubes. We developed a microplate PIXUL-based TRIzol-free and column-free RNA isolation protocol that uses a buffer containing proteinase K (PK buffer). We have integrated the CryoGrid system with PIXUL-based PK buffer, TRIzol, and PureLink column methods to isolate RNA for gene-specific qPCR and genome-wide transcript analyses. CryoGrid-PIXUL, when integrated with either PK buffer, TRIzol or PureLink column RNA isolation protocols, yielded similar transcript profiles in frozen organs (brain, heart, kidney and liver) from a mouse model of sepsis.ConclusionsRNA isolation using the CryoGrid-PIXUL system combined with the 96-well microplate PK buffer method offers an inexpensive user-friendly high throughput workflow to study transcriptional responses in tissues in health and disease as well as in therapeutic interventions.

Vision for the 12 LABOURS Digital Twin Platform.

Clinical translation of personalised computational physiology workflows and digital twins can revolutionise healthcare by providing a better understanding of an individual's physiological processes and any changes that could lead to serious health consequences. However, the lack of common infrastructure for developing these workflows and digital twins has hampered the realisation of this vision. The Auckland Bioengineering Institute's 12 LABOURS project aims to address these challenges by developing a Digital Twin Platform to enable researchers to develop and personalise computational physiology models to an individual's health data in clinical workflows. This will allow clinical trials to be more efficiently conducted to demonstrate the efficacy of these personalised clinical workflows. We present a demonstration of the platform's capabilities using publicly available data and an existing automated computational physiology workflow developed to assist clinicians with diagnosing and treating breast cancer. We also demonstrate how the platform facilitates the discovery and exploration of data and the presentation of workflow results as part of clinical reports through a web portal. Future developments will involve integrating the platform with health systems and remote-monitoring devices such as wearables and implantables to support home-based healthcare. Integrating outputs from multiple workflows that are applied to the same individual's health data will also enable the generation of their personalised digital twin.Clinical Relevance- The proposed 12 LABOURS Digital Twin Platform will enable researchers to 1) more efficiently conduct clinical trials to assess the efficacy of their computational physiology workflows and support the clinical translation of their research; 2) reuse primary and derived data from these workflows to generate novel workflows; and 3) generate personalised digital twins by integrating the outputs of different computational physiology workflows.