Complex Scientific Workflows Research Articles

Input/output optimization scheduler for cloud-based map reduce framework

Hadoop MapReduce (HMR) provides the most common MapReduce (MR) framework, and it is available as open source. MR is a famous computational framework for evaluating unstructured, and semi-structured big data and executing applications in the past ten years. Memory and input/output (I/O) overhead are just two of the many problems affecting the current HMR scheduler system. This study aims to improve systems resource use including the processing of data in real-time by creating a memory I/O optimized scheduler (MIOOS) for HMR. The disk I/O seek can be reduced by using MIOOS, which analyzes the entire memory management. Additionally, the MIOOS makespan approach is used to reduce the occurrence of problems in intermediary tasks. Both the MIOOS approach and the current approach are assessed by using complex scientific workflow applications with extreme task inter-dependencies. Further, the comparison study demonstrates that the MIOOS framework outdoes the current approach regarding makespan and overall memory usage.

A review on workflow scheduling and resource allocation algorithms in distributed mobile clouds

AbstractThe advent of distributed computing and mobile clouds made it possible to transfer and distribute the heavy processes of complex workflows to the cloud. Managing and with the help of the mobile cloud, virtualization and shared resources managing the executing large‐scale workflows is possible. Since virtual resources are used, scheduling and resource allocation are important and key research topics in public cloud environment and mobile clouds. The cloud model of mobile devices with public or virtual cloud, which consists of temporary mobile devices, is an interesting topic to be investigated. Workflow scheduling and availability of resources in mobile clouds by high mobility and Energy efficient, it is one of the challenges and problems that must be investigated one of the main challenges in this research area. The purpose of scheduling algorithms is to improve service quality criteria by observing the constraints. In this article, the main goal is to extensively review the scheduling algorithms for the complex scientific workflow in public and mobile clouds. Furthermore, a review of the use of heuristic and meta‐heuristic techniques in dynamic and static states, with various constraints for task scheduling in cloud and mobile computing has been performed. This article accomplishes a review of using existing techniques for scheduling tasks in mobile clouds. We also present a comprehensive analysis and systematic comparison between these scheduling algorithms.

MONWS: Multi-Objective Normalization Workflow Scheduling for Cloud Computing

Cloud computing is a prominent approach for complex scientific and business workflow applications in the pay-as-you-go model. Workflow scheduling poses a challenge in cloud computing due to its widespread applications in physics, astronomy, bioinformatics, and healthcare, etc. Resource allocation for workflow scheduling is problematic due to the computationally intensive nature of the workflow, the interdependence of tasks, and the heterogeneity of cloud resources. During resource allocation, the time and cost of execution are significant issues in the cloud-computing environment, which can potentially degrade the service quality that is provided to end users. This study proposes a method focusing on makespan, average utilization, and cost. The authors propose a task’s dynamic priority for workflow scheduling using MONWS, which uses the min-max algorithm to minimize the finish time and maximize resource utilization by calculating the dynamic threshold value for scheduling tasks on virtual machines. When the experimental results were compared to existing algorithms, MONWS achieved a 35% improvement in makespan, an 8% increase in maximum average cloud utilization, and a 4% decrease in cost.

AILS: A budget-constrained adaptive iterated local search for workflow scheduling in cloud environment

With the rapid development of cloud computing, scheduling the complex scientific workflow on the cloud becomes an extraordinarily challenging problem. Especially, the workflow scheduling in cloud environment often need to meet several constraints such as budget. To reduce the execution time subject to a specific budget, this paper proposes a novel adaptive iterated local search framework (AILS). In the proposed AILS, a greedy resource provisioning scheme and HEFT are combined to generate the initial solution. Based on a budget-aware diversification strategy and level-based order perturbation strategy, a perturbation mechanism is applied to perform exploration. In addition, a novel intensification strategy is utilized to execute exploitation. Furthermore, an adaptive penalty function is employed to guide the search near the bound of feasible solution space. Based on the components of AILS, the Markov chain is employed to analyze the convergence of AILS. A comprehensive comparison is constructed to evaluate the proposed AILS, the simulating results indicate that the average relative percentage deviation of AILS is 3.6% lower than that of GRP-HEFT. Compared with the meta-heuristics, the AILS is effective and competitive for the considered problem.

Fault tolerance for a scientific workflow system in a Cloud computing environment

High computation power is required to execute complex scientific workflows. Cloud computing resources are used viably to perform such complex workflows. Task clustering has demonstrated to be an efficient technique to decrease system overhead and to enhance the fine computational granularity tasks of a scientific workflow executed on distributed resources. However, earlier clustering methods ignore the effect of failures on the system, despite their significant impact on large-scale distributed resources, such as Clouds. In this paper, we present a new fault-tolerant task clustering method called FT-HCC that is designed by including the workflow execution time (makespan) and execution cost constraints, which are used to increase workflow performance. The proposed method is implemented and evaluated in a simulation-based approach, using a real-time workflow execution to analyze performance improvement. The results consolidate that the proposed strategies and techniques work efficiently in terms of fault tolerance and improve both workflow makespan and execution cost when compared to existing approaches.

Implementing efficient data compression and encryption in a persistent key-value store for HPC

Recently, persistent data structures, like key-value stores (KVSs), which are stored in a high-performance computing (HPC) system’s nonvolatile memory, provide an attractive solution for a number of emerging challenges like limited I/O performance. Data compression and encryption are two well-known techniques for improving several properties of such data-oriented systems. This article investigates how to efficiently integrate data compression and encryption into persistent KVSs for HPC with the ultimate goal of hiding their costs and complexity in terms of performance and ease of use. Our compression technique exploits deep memory hierarchy in an HPC system to achieve both storage reduction and performance improvement. Our encryption technique provides a practical level of security and enables sharing of sensitive data securely in complex scientific workflows with nearly imperceptible cost. We implement the proposed techniques on top of a distributed embedded KVS to evaluate the benefits and costs of incorporating these capabilities along different points in the dataflow path, illustrating differences in effective bandwidth, latency, and additional computational expense on Swiss National Supercomputing Centre’s Grand Tavé and National Energy Research Scientific Computing Center’s Cori.

Development of cognitive fault tolerant model for scientific workflows by integrating overlapped migration and check-pointing approach

High computation power and storage space are needed to execute the complex scientific workflows. Cloud computing resources are used effectively to execute such complex workflows. Tasks executing in the workflow is dependent in nature and hence failure of a task affects the overall performance of the execution. In order to execute the workflow without any interrupt, a proactive intelligent fault tolerant model is necessary. This paper proposes a model called cognitive fault tolerant (CFT) with three important phases for tolerating the task and VM failure proactively. In prediction phase, combined ensemble prediction method is used to predict the task failures and label tuning algorithms are used to generate the intermediate labels and to strengthen the prediction. The segregation phase isolates the task based on priority assignment. Last phase of CFT model is recovery. Fitness checking is used to find whether the predicted failure is due to task or VM. Post prediction checkpointing (PPC) method is used as recovery process for task failure. VM failure can be recovered using post or pre replication overlapped migration method. The validation of proposed CFT model can be done by comparing with other existing algorithms. Experimental analysis shows that proposed CFT model improvise the reliability of workflow execution in cloud environment.

Supercomputers, Clouds and Grids powered by BigPanDA for Brain studies

The PanDA WMS - Production and Distributed Analysis Workload Management System - has been developed and used by the ATLAS experiment at the LHC (Large Hadron Collider) for all data processing and analysis challenges. BigPanDA is an extension of the PanDA WMS to run ATLAS and non-ATLAS applications on Leadership Class Facilities and supercomputers, as well as traditional grid and cloud resources. The success of the BigPanDA project has drawn attention from other compute intensive sciences such as biology. In 2017, a pilot project was started between BigPanDA and the Blue Brain Project (BBP) of the Ecole Polytechnique Federal de Lausanne (EPFL) located in Lausanne, Switzerland. This proof of concept project is aimed at demonstrating the efficient application of the BigPanDA system to support the complex scientific workflow of the BBP, which relies on using a mix of desktop, cluster and supercomputers to reconstruct and simulate accurate models of brain tissue.

Building a scientific workflow framework to enable real‐time machine learning and visualization

SummaryNowadays, we have entered the era of big data. In the area of high performance computing, large‐scale simulations can generate huge amounts of data with potentially critical information. However, these data are usually saved in intermediate files and are not instantly visible until advanced data analytics techniques are applied after reading all simulation data from persistent storages (eg, local disks or a parallel file system). This approach puts users in a situation where they spend long time on waiting for running simulations while not knowing the status of the running job. In this paper, we build a new computational framework to couple scientific simulations with multi‐step machine learning processes and in‐situ data visualizations. We also design a new scalable simulation‐time clustering algorithm to automatically detect fluid flow anomalies. This computational framework is built upon different software components and provides plug‐in data analysis and visualization functions over complex scientific workflows. With this advanced framework, users can monitor and get real‐time notifications of special patterns or anomalies from ongoing extreme‐scale turbulent flow simulations.

A hyper-heuristic cost optimisation approach for Scientific Workflow Scheduling in cloud computing

Effective management of Scientific Workflow Scheduling (SWFS) processes in a cloud environment remains a challenging task when dealing with large and complex Scientific Workflow Applications (SWFAs). Cost optimisation of SWFS benefits cloud service consumers and providers by reducing temporal and monetary costs in processing SWFAs. However, cost optimisation performance of SWFS approaches is affected by the inherent nature of the SWFA as well as various types of scenarios that depend on the number of available virtual machines and varied sizes of SWFA datasets. Cost optimisation performance of existing SWFS approaches is still not satisfactory for all considered scenarios. Thus, there is a need to propose a dynamic hyper-heuristic approach that can effectively optimise the cost of SWFS for all different scenarios. This can be done by employing different meta-heuristic algorithms in order to utilise their strengths for each scenario. Thus, the main objective of this paper is to propose a Completion Time Driven Hyper-Heuristic (CTDHH) approach for cost optimisation of SWFS in a cloud environment. The CTDHH approach employs four well-known population-based meta-heuristic algorithms, which act as Low Level Heuristic (LLH) algorithms. In addition, the CTDHH approach enhances the native random selection way of existing hyper-heuristic approaches by incorporating the best computed workflow completion time to act as a high-level selector to dynamically pick a suitable algorithm from the pool of LLH algorithms after each run. A real-world cloud based experimentation environment has been considered to evaluate the performance of the proposed CTDHH approach by comparing it with five baseline approaches, i.e. four population-based approaches and an existing hyper-heuristic approach named Hyper-Heuristic Scheduling Algorithm (HHSA). Several different scenarios have also been considered to evaluate data-intensiveness and computation-intensive performance. Based on the results of the experimental comparison, the proposed approach has proven to yield the most effective performance results for all considered experimental scenarios.

Time and Energy Optimization Algorithms for the Static Scheduling of Multiple Workflows in Heterogeneous Computing System

The heterogeneous computing system (HCS), which is used to deal with complex and enormous business or scientific workflows, is playing a very important role as cloud computing rapidly develops. For multiple workflows computing in HCS, one of challenging issues is how to make a reasonable tradeoff between the schedule length and energy consumption. In this paper, we focus on a workflow that can be represented by a directed acyclic graph (DAG). We propose the corresponding algorithms which cooperate with dynamic voltage and frequency scaling (DVFS) technique to address the aforementioned concern and evaluate the algorithms in terms of randomly generated DAGs, real application DAGs and their hybrids under DVFS-enabled HCS. From the experimental results, we draw the conclusion that interleaving workflows lead to a better average tradeoff when scheduling multiple workflows in HCS.

PANORAMA: An approach to performance modeling and diagnosis of extreme-scale workflows

Computational science is well established as the third pillar of scientific discovery and is on par with experimentation and theory. However, as we move closer toward the ability to execute exascale calculations and process the ensuing extreme-scale amounts of data produced by both experiments and computations alike, the complexity of managing the compute and data analysis tasks has grown beyond the capabilities of domain scientists. Thus, workflow management systems are absolutely necessary to ensure current and future scientific discoveries. A key research question for these workflow management systems concerns the performance optimization of complex calculation and data analysis tasks. The central contribution of this article is a description of the PANORAMA approach for modeling and diagnosing the run-time performance of complex scientific workflows. This approach integrates extreme-scale systems testbed experimentation, structured analytical modeling, and parallel systems simulation into a comprehensive workflow framework called Pegasus for understanding and improving the overall performance of complex scientific workflows.

Data Management Challenges in Next Generation Sequencing

Since the early days of the Human Genome Project, data management has been recognized as a key challenge for modern molecular biology research. By the end of the nineties, technologies had been established that adequately supported most ongoing projects, typically built upon relational database management systems. However, recent years have seen a dramatic increase in the amount of data produced by typical projects in this domain. While it took more than ten years, approximately three billion USD, and more than 200 groups worldwide to assemble the first human genome, today’s sequencing machines produce the same amount of raw data within a week, at a cost of approximately 2000 USD, and on a single device. Several national and international projects now deal with (tens of) thousands of genomes, and trends like personalized medicine call for efforts to sequence entire populations. In this paper, we highlight challenges that emerge from this flood of data, such as parallelization of algorithms, compression of genomic sequences, and cloud-based execution of complex scientific workflows. We also point to a number of further challenges that lie ahead due to the increasing demand for translational medicine, i.e., the accelerated transition of biomedical research results into medical practice.

CMS Workflow Execution Using Intelligent Job Scheduling and Data Access Strategies

Complex scientific workflows can process large amounts of data using thousands of tasks. The turnaround times of these workflows are often affected by various latencies such as the resource discovery, scheduling and data access latencies for the individual workflow processes or actors. Minimizing these latencies will improve the overall execution time of a workflow and thus lead to a more efficient and robust processing environment. In this paper, we propose a pilot job based infrastructure that has intelligent data reuse and job execution strategies to minimize the scheduling, queuing, execution and data access latencies. The results have shown that significant improvements in the overall turnaround time of a workflow can be achieved with this approach. The proposed approach has been evaluated, first using the CMS Tier0 data processing workflow, and then simulating the workflows to evaluate its effectiveness in a controlled environment.

Fault Tolerance In Grid Computing: State of the Art and Open Issues

Fault tolerance is an important property for large scale computational grid systems, where geographically distributed nodes co-operate to execute a task. In order to achieve high level of reliability and availability, the grid infrastructure should be a foolproof fault tolerant. Since the failure of resources affects job execution fatally, fault tolerance service is essential to satisfy QOS requirement in grid computing. Commonly utilized techniques for providing fault tolerance are job checkpointing and replication. Both techniques mitigate the amount of work lost due to changing system availability but can introduce significant runtime overhead. The latter largely depends on the length of checkpointing interval and the chosen number of replicas, respectively. In case of complex scientific workflows where tasks can execute in well defined order reliability is another biggest challenge because of the unreliable nature of the grid resources.

A grid services cloud for molecular modelling workflows

Scientific workflows require seamless access to HPC applications, deployed on remote, globally distributed computing resources. Typically, scientific workflows are both compute- and data-intensive, and often require dynamic execution control mechanisms. We present a service-oriented infrastructure that addresses these challenges by seamlessly integrating grid computing technologies with a Cloud infrastructure to support the scheduling of dynamic scientific workflows. A case study implementing a complex scientific workflow for computing photodynamics of biologically relevant molecules, a simulation of the non-adiabatic dynamics of 2,4-pentadieneiminum-cation (Protonated Schiff Base 3 (PSB3)) solvated in Water, is realised via the presented infrastructure.

A workflow model for heterogeneous computing environments

This work presents the details of a model for fully decentralised scientific workflow management designed to support the efficient execution of applications in heterogeneous and unpredictable computing environments. The proposed model involves a novel application of the overlay metacomputer concept, built on a graph-oriented peer to peer framework. The use of peer to peer overlay on both batch and cycle scavenging systems simultaneously enables the aggregation and virtual homogenisation of heterogeneous resource collectives, and makes possible the execution of complex scientific workflow applications on otherwise batch oriented processors. Overlaying a peer to peer workflow management system on multiple independent sites decouples task expression from resource allocation, i.e., the host resource managers allocate processing elements to host the peers, allowing the peers execute multiple workflow tasks. This facilitates the efficient execution of very fine grained workflow expressions, which is shown to reduce both application makespan and wasted cycles when using processing elements allocated by a pre-emptive resource manager. Quantitative results are presented based on performance observed in a large scale heterogeneous multi-user environment.

A throughput maximization strategy for scheduling transaction‐intensive workflows on SwinDeW‐G

AbstractWith the rapid development of e‐business, workflow systems now have to deal with transaction‐intensive workflows whose main characteristic is the huge number of concurrent workflow instances. For such workflows, it is important to maximize the overall throughput to provide good quality of service. However, most of the existing scheduling algorithms are designed for scheduling of a single complex scientific workflow instance and are not efficient enough for scheduling transaction‐intensive workflows. To address this problem, we propose a throughput maximization strategy (TMS), which contains two specific algorithms for scheduling transaction‐intensive workflows at the instance and task levels, respectively. The first algorithm called Opposite Average Load tries to maximize the overall throughput by pursuing the overall load balance at the instance level, whereas the second algorithm called Extended Min–Min tries to further maximize the overall throughput at the task level by increasing the utilization rate of resources within each local autonomous group. The comparison and simulation performed on Swinburne Decentralized Workflow for Grid (SwinDeW‐G), a peer‐to‐peer‐based grid workflow environment, demonstrate that our strategy can improve the overall throughput significantly over existing scheduling algorithms when scheduling transaction‐intensive workflows. Copyright © 2008 John Wiley & Sons, Ltd.

IQ-Paths: Predictably High Performance Data Streams Across Dynamic Network Overlays

Overlay networks are a key vehicle for delivering network and processing resources to high performance applications. For shared networks, however, to consistently deliver such resources at desired levels of performance, overlays must be managed at runtime, based on the continuous assessment and prediction of available distributed resources. Data-intensive applications, for example, must assess, predict, and judiciously use available network paths, and dynamically choose alternate or exploit concurrent paths. Otherwise, they cannot sustain the consistent levels of performance required by tasks like remote data visualization, online program steering, and remote access to high end devices. The multiplicity of data streams occurring in complex scientific workflows or in large-scale distributed collaborations exacerbate this problem, particularly when different streams have different performance requirements. This paper presents IQ-Paths, a set of techniques and their middleware realization that implement self-regulating overlay streams for data-intensive distributed applications. Self-regulation is based on (1) the dynamic and continuous assessment of the quality of each overlay path, (2) the use of online network monitoring and statistical analyses that provide probabilistic guarantees about available path bandwidth, loss rate, and RTT, and (3) self-management, via an efficient packet routing and scheduling algorithm that dynamically schedules data packets to different overlay paths in accordance with their available bandwidths. IQ-Paths offers probabilistic guarantees for application-level specifications of stream utility, based on statistical predictions of available network bandwidth. This affords applications with the ability, for instance, to send control or steering data across overlay paths that offer strong guarantees for future bandwidth vs. across less guaranteed paths. Experimental results presented in this paper use IQ-Paths to better handle the different kinds of data produced by two high performance applications and one multimedia application: (1) a data-driven interactive high performance code with user-defined utility requirements, (2) an adaptive overlay version of the popular Grid-FTP application, and (3) a MPEG-4 Fine-Grained Scalable layered video streaming.

Grid Service Orchestration Using the Business Process Execution Language (BPEL)

Modern scientific applications often need to be distributed across Grids. Increasingly applications rely on services, such as job submission, data transfer or data portal services. We refer to such services as Grid services. While the invocation of Grid services could be hard coded in theory, scientific users want to orchestrate service invocations more flexibly. In enterprise applications, the orchestration of web services is achieved using emerging orchestration standards, most notably the Business Process Execution Language (BPEL). We describe our experience in orchestrating scientific workflows using BPEL. We have gained this experience during an extensive case study that orchestrates Grid services for the automation of a polymorph prediction application. Using this example, we explain the extent with which the BPEL language supports the definition of scientific workflows. We then describe the reliability, performance and scalability that can be achieved by executing a complex scientific workflow with ActiveBPEL, an industrial strength but freely available BPEL engine.