Parallel Workflow Research Articles

Abstract 96: Assessment of the Rapid Arterial oCclusion Evaluation (RACE) Scale in Real-World Practice for Prediction of Large Vessel Occlusion and Reducing Time to Thrombectomy

Background: Pre-hospital identification of potential large vessel occlusion (LVO) stroke patients may lead to faster triage and treatment. We examined whether the RACE scale can be reliably implemented in a real-world setting with multiple EMS agencies and lead to rapid treatment. Methods: A prospective study was performed at a high volume comprehensive stroke center. In the first phase, eight EMS agencies were educated on use of the RACE scale using an online training video. All EMS stroke alerts were recorded. When EMS RACE score was 5 or higher, the neurocath lab team was alerted prior to EMS arrival as part of a parallel workflow. Upon ER arrival, the following characteristics were tracked: NIHSS score, RACE score, CT findings, presence of LVO and workflow time metrics. Results: During the study period (January 2016 to June 2017), RACE score was provided for 797 of 1498 EMS stroke alerts (53%). Higher pre-hospital RACE scores correlated with NIHSS scores. LVO was found in 106 (13%) of patients with an available RACE score. A RACE score of 5 or higher was able to identify 64% of all LVO patients (sensitivity: 64%; specificity: 72%; PPV: 30%; NPV: 93%; accuracy: 71%; Youden’s index). However, of the 260 patients with RACE score 5 or higher, only 68 patients (26%) were found to have LVO while 29 patients (11%) had ICH (figure); among 499 patients with RACE score less than 5, LVO was present in 38 patients (8%). When an EMS stroke alert with high RACE score triggered early alert of the neurocath lab team, median door to groin puncture time for thrombectomy was 65 minutes compared to 91 minutes for cases with sequential workflow. Conclusion: The RACE scale can be successfully implemented across EMS agencies and results in faster door to groin puncture times. While a RACE score of 5 or higher is associated with greater likelihood of LVO, there are a significant number of false positives. Further refinement of pre-hospital stroke severity scales is warranted to improve the accuracy of this approach.

Full waveform inversion: early start of model building process to resolve the complex near surface model in the Exmouth sub-basin, North West Shelf Australia

The Schlumberger Multiclient Exmouth 3D survey was acquired over the Exmouth sub-basin, North West Shelf Australia and covers 12 600 km2. One of the primary objectives of this survey was to produce a wide coverage of high quality imaging with advanced processing technology within an agreed turnaround time. The complexity of the overburden was one of the imaging challenges that impacted the structuration and image quality at the reservoir level. Unlike traditional full-waveform inversion (FWI) workflow, here, FWI was introduced early in the workflow in parallel with acquisition and preprocessing to produce a reliable near surface velocity model from a smooth starting model. FWI derived an accurate and detailed near surface model, which subsequently benefitted the common image point (CIP) tomography model updates through to the deeper intervals. The objective was to complete the FWI model update for the overburden concurrently with the demultiple stages hence reflection time CIP tomography could start with a reasonably good velocity model upon completion of the demultiple process.

ExSeisDat: A set of parallel I/O and workflow libraries for petroleum seismology

Seismic data-sets are extremely large and are broken into data files, ranging in size from 100s of GiBs to 10s of TiBs and larger. The parallel I/O for these files is complex due to the amount of data along with varied and multiple access patterns within individual files. Properties of legacy file formats, such as the de-facto standard SEG-Y, also contribute to the decrease in developer productivity while working with these files. SEG-Y files embed their own internal layout which could lead to conflict with traditional, file-system-level layout optimization schemes. Additionally, as seismic files continue to increase in size, memory bottlenecks will be exacerbated, resulting in the need for smart I/O optimization not only to increase the efficiency of read/writes, but to manage memory usage as well. The ExSeisDat (Extreme-Scale Seismic Data) set of libraries addresses these problems through the development and implementation of easy to use, object oriented libraries that are portable and open source with bindings available in multiple languages. The lower level parallel I/O library, ExSeisPIOL (Extreme-Scale Seismic Parallel I/O Library), targets SEG-Y and other proprietary formats, simplifying I/O by internally interfacing MPI-I/O and other I/O interfaces. The I/O is explicitly handled; end users only need to define the memory limits, decomposition of I/O across processes, and data access patterns when reading and writing data. ExSeisPIOL bridges the layout gap between the SEG-Y file structure and file system organization. The higher level parallel seismic workflow library, ExSeisFlow (Extreme-Scale Seismic workFlow), leverages ExSeisPIOL, further simplifying I/O by implicitly handling all I/O parameters, thus allowing geophysicists to focus on domain-specific development. Operations in ExSeisFlow focus on prestack processing and can be performed on single traces, individual gathers, and across entire surveys, including out of core sorting, binning, filtering, and transforming. To optimize memory management, the workflow only reads in data pertinent to the operations being performed instead of an entire file. A smart caching system manages the read data, discarding it when no longer needed in the workflow. As the libraries are optimized to handle spatial and temporal locality, they are a natural fit to burst buffer technologies, particularly DDN’s Infinite Memory Engine (IME) system. With appropriate access semantics or through the direct exploitation of the low-level interfaces, the ExSeisDat stack on IME delivers a significant improvement to I/O performance over standalone parallel file systems like Lustre.

Predicting temporal violations for parallel business cloud workflows

SummaryWorkflow temporal violations, namely, intermediate workflow runtime delays, often occur and have a serious impact on the on‐time completion of massive concurrent requests. Therefore, accurate prediction of cloud workflow temporal violations is critical as its result can serve as an essential reference for temporal violation prevention and handling strategies. Conventional studies mainly focus on the time delays of a single workflow activity or a single workflow instance but overlook the propagation of time delays among them. This is a serious problem as time delays can propagate in cloud workflow system due to resource sharing and the dependencies among workflow activities. This paper first proposes a novel temporal violation transmission model inspired by an epidemic model to model the dynamics of time delay propagation. Afterward, a novel temporal violation prediction strategy is presented to estimate the number of temporal violations that may occur and determine the number of violations that must be handled to achieve the target service‐level agreement, namely, the on‐time completion rate. To the best of our knowledge, this is the first attempt to predict cloud workflow temporal violations at the workflow build‐time stage by analyzing the propagation of temporal violations. Experimental results demonstrate that our strategy can make highly accurate predictions and is scalable for a large batch of parallel workflows running in the cloud.

Improving mechanical thrombectomy time metrics in the angiography suite: Stroke cart, parallel workflows, and conscious sedation.

Purpose Earlier reperfusion of large-vessel occlusion (LVO) stroke improves functional outcomes. We hypothesize that use of a stroke cart in the angiography suite, containing all commonly used procedural equipment in a mechanical thrombectomy, combined with parallel staff workflows, and use of conscious sedation when possible, improve mechanical thrombectomy time metrics. Methods We identified 47 consecutive LVO patients who underwent mechanical thrombectomy at our center, retrospectively and prospectively from implementation of these three workflow changes (19 pre- and 28 post-). For each patient, last known normal, NIHSS, angiography suite in-room time, type of anesthesia, groin puncture time, on-clot time, recanalization time, LVO location, number of passes, device(s) used, mTICI score, and outcome (mRS) were recorded. Between-group comparisons of time metrics and multivariate regression were performed. Results Stroke cart, parallel workflows, and primary use of conscious sedation decreased in-room time to groin puncture (-21.3 min, p < 0.0001), in-room to on-clot time (-24.1 min, p = 0.001), and in-room to reperfusion time (-29.5 min, p = 0.01). In a multivariate analysis, endotracheal intubation and general anesthesia were found to significantly increase in-room to on-clot time ( p = 0.01), in-room to reperfusion time ( p = 0.01), and groin puncture to on-clot time ( p = 0.05). The number of patients achieving a good outcome (mRS 0-2), however, did not significantly differ between the two groups (9/18 (47%) vs 14/28 (50%), p = 0.60). Conclusions Use of a stroke cart, parallel workflows by neurointerventionalists, technologists, and nursing staff, and use of conscious sedation may be useful to other institutions in efforts to improve procedural times.

Orchid: a novel management, annotation and machine learning framework for analyzing cancer mutations.

As whole-genome tumor sequence and biological annotation datasets grow in size, number and content, there is an increasing basic science and clinical need for efficient and accurate data management and analysis software. With the emergence of increasingly sophisticated data stores, execution environments and machine learning algorithms, there is also a need for the integration of functionality across frameworks. We present orchid, a python based software package for the management, annotation and machine learning of cancer mutations. Building on technologies of parallel workflow execution, in-memory database storage and machine learning analytics, orchid efficiently handles millions of mutations and hundreds of features in an easy-to-use manner. We describe the implementation of orchid and demonstrate its ability to distinguish tissue of origin in 12 tumor types based on 339 features using a random forest classifier. Orchid and our annotated tumor mutation database are freely available at https://github.com/wittelab/orchid. Software is implemented in python 2.7, and makes use of MySQL or MemSQL databases. Groovy 2.4.5 is optionally required for parallel workflow execution. JWitte@ucsf.edu. Supplementary data are available at Bioinformatics online.

Towards a continuous inland excess water flood monitoring system based on remote sensing data

Abstract Inland excess water (IEW) is a type of flood where large flat inland areas are covered with water during a period of several weeks to months. The monitoring of these floods is needed to understand the extent and direction of development of the inundations and to mitigate their damage to the agricultural sector and build up infrastructure. Since IEW affects large areas, remote sensing data and methods are promising technologies to map these floods. This study presents the first results of a system that can monitor inland excess water over a large area with sufficient detail at a high interval and in a timely matter. The methodology is developed in such a way that only freely available satellite imagery is required and a map with known water bodies is needed to train the method to identify inundations. Minimal human interference is needed to generate the IEW maps. We will present a method describing three parallel workflows, each generating separate maps. The maps are combined to one weekly IEW map. At this moment, the method is capable of generating IEW maps for a region of over 8000 km2, but it will be extended to cover the whole Great Hungarian Plain, and in the future, it can be extended to any area where a training water map can be created.

SAVI objects: sharing and virtuality incorporated

Direct sharing and storing of memory objects allows high-performance and low-overhead collaboration between parallel processes or application workflows with loosely coupled programs. However, sharing of objects is hindered by the inability to use subtype polymorphism which is common in object-oriented programming languages. That is because implementations of subtype polymorphism in modern compilers rely on using virtual tables stored at process-specific locations, which makes objects unusable in processes other than the creating process. In this paper, we present SAVI Objects, objects with Sharing and Virtuality Incorporated. SAVI Objects support subtype polymorphism but can still be shared across processes and stored in persistent data structures. We propose two different techniques to implement SAVI Objects and evaluate the tradeoffs between them. The first technique is virtual table duplication which adheres to the virtual-table-based implementation of subtype polymorphism, but duplicates virtual tables for shared objects to fixed memory addresses associated with each shared memory region. The second technique is hashing-based dynamic dispatch which re-implements subtype polymorphism using hashing-based look-ups to a global virtual table. Our results show that SAVI Objects enable direct sharing and storing of memory objects that use subtype polymorphism by adding modest overhead costs to object construction and dynamic dispatch time. SAVI Objects thus enable faster inter-process communication, improving the overall performance of production applications that share polymorphic objects.

LIMS configuration to fit new massively parallel sequencing workflows in forensic genetics

Laboratory Information Management Systems (LIMS) are valuable tools in the forensic genetics laboratory, streamlining sample and analysis traceability, batching of multiple samples, integration and data transfer with genetic analyzers, monitoring of test results, DNA data exchange with National DNA databases, and flexible reporting. This work, developed in the framework of the DNASEQEX project, presents the configuration of a commercial LIMS system to fit the automated workflow provided by the IonChef/Ion S5XL platforms and the Torrent Server Suite/Converge software packages for the massively parallel sequence analysis of STR markers in the forensic lab. LIMS configurations included (1) the setup of analysis and batch templates for DNA library preparation, and DNA library quantification (2) the creation of composite samples for MPS-STR library tracking, (3) the development of files to electronically export MPS sample setups and analysis to the Torrent Server Software, (4) the configuration of scripts for the electronic transfer into the LIMS system of STR allele and sequence data files generated by the Converge software, and (5)the development of DNA data export files to national DNA databases. The developed LIMS configuration provides the laboratory with the tools required for setting up automated MPS workflows, helping to efficiently and accurately acquire, manage, and exchange MPS data in the forensic genetics laboratory.

Comparison of digital PCR with real-time PCR calibrated to the International Scale for BCR-ABL monitoring.

e18545 Background: Monitoring BCR-ABL1 fusion transcripts is the cornerstone of management in chronic myeloid leukemia (CML). Real Time PCR(RTPCR) has been the tool of choice for its sensitivity and dynamic range. The International Scale was recently introduced to allow for standardization across laboratories. Chip based Digital PCR(dPCR) which allows for absolute quantitation may obliviate the need for such standardization by giving absolute results.It has the potential to increase sensitivity for detection of minimal residual disease. This assumes great significance as we move into the era of treatment free remission. Methods: A total of 31 EDTA-blood samples of CML patients with known values ranging from 0.003-0.5 IS were processed via an accredited RTPCR protocol calibrated to the IS scale and a parallel dPCR workflow without any specific calibration . The analysis of digital PCR samples was performed on the Thermofisher Cloud Platform. Results: Both RTPCR and dPCR yielded extremely concordant results. The Pearson correlation between the two methods was r = 0.6043 (95%CI: 0.318 to 0.789; p = 0.0003). The calculated BCR-ABL/ABL ratios were comparable with a median of 0.098 for RT-PCR calibrated to the IS (range 0.003-0.55; n = 26) and 0.11 for dPCR (range 0.003-0.37; n = 29). The mean difference for the ratios between the two methods used for the detection was around 0.08. Conclusions: We demonstrate here the capability of dPCR to provide a parallel result to an IS scale calibrated protocol without any standardization. This approach required no specific calibrators or standards and resulted in extremely cost effective testing. This freedom from routine calibration provides for a significantly more robust workflow and greatly increased reliability. Limitations do persist in dPCR on account of limited chip densities which are the main parameter for the Poisson statistics. This has limited the dynamic range on dPCR to a maximum of Log 4 for accurate quantification. However as chip densities and emulsion concentrations increase in these technologies, they are poised to introduce a new era in the quest of accurate quantification.

Multi-objective workflow scheduling in cloud system based on cooperative multi-swarm optimization algorithm

In order to improve the performance of multi-objective workflow scheduling in cloud system, a multi-swarm multiobjective optimization algorithm (MSMOOA) is proposed to satisfy multiple conflicting objectives. Inspired by division of the same species into multiple swarms for different objectives and information sharing among these swarms in nature, each physical machine in the data center is considered a swarm and employs improved multi-objective particle swarm optimization to find out non-dominated solutions with one objective in MSMOOA. The particles in each swarm are divided into two classes and adopt different strategies to evolve cooperatively. One class of particles can communicate with several swarms simultaneously to promote the information sharing among swarms and the other class of particles can only exchange information with the particles located in the same swarm. Furthermore, in order to avoid the influence by the elastic available resources, a manager server is adopted in the cloud data center to collect the available resources for scheduling. The quality of the proposed method with other related approaches is evaluated by using hybrid and parallel workflow applications. The experiment results highlight the better performance of the MSMOOA than that of compared algorithms.

Abstract 050: Improvements in Door to Groin Puncture Time for Surgical Stroke After Quality Protocol Interventions at the University of California, San Diego

Introduction: Delays in door to groin puncture time (DGPT) for patients with ischemic stroke caused by acute large vessel occlusion (LVO) correlate with worse clinical outcomes. Stroke centers aim to minimize DGPT to facilitate prompt intervention and limit ischemic brain injury. In this study, we present the results of a comprehensive quality assessment at the University of California, San Diego (UCSD). From 2015 to 2016, institutional implementation of a quality improvement protocol significantly reduced DGPT. Materials and Methods: Beginning July 2015, the UCSD interdisciplinary stroke team implemented a series of quality improvement measures to decrease DGPT, with a target of 90 minutes or less. After each case, areas of inefficiency were identified and changes were implemented based on direct feedback from neurointerventional physicians and ancillary staff. Changes included: 1) creation of a pager group notification system to activate the entire neurointerventional team simultaneously, 2) consistently involving anesthesia with each neurointervention, 3) streamlining communication between the vascular neurology and neurointervention teams, and 4) structuring parallel workflows to enhance mobilization speed. R statistical software was utilized to compare DGPT before and after implementation of these process improvements. Patients were divided into three groups based on the date of their intervention as follows: 23 patients treated from July-December 2015, 24 patients treated from January-July 2016, and 14 patients treated from July 2016-December 2016. A multivariable univariate binary logistic regression model was constructed to capture predictors of compliance with our target DGPT (&lt;90 min). Variables analyzed included: date of intervention, mode of patient admission (i.e. transfer, direct admit from ED, inpatient), hospital location, age, and gender. Results: 61 patients underwent mechanical thrombectomy for treatment of acute LVO from July 2015 to December 2016. In our analysis, date of intervention—as a proxy for implementation of process improvement protocols—and mode of admission were predictive of compliance with target DGPT. Patients who were treated from July 2016 to December 2016—after full implementation of process improvements— were 9.5 times more likely to meet or exceed the target DGPT compared to patients treated July 2015 to December 2015 (p=0.01). Additionally, arrival via transfer from an outside hospital was determined to be an independent predictor of meeting DGPT goals. (p=0.02). Conclusion: UCSD’s quality improvement process effected dramatic, statistically significant improvement in DGPT. This analysis demonstrates the utility of a formal quality improvement system at a large, academic comprehensive stroke center.

Abstract TP281: Reducing Door-to-Needle Times in Acute Ischemic Stroke : Multidisciplinary Team-based Approach at a Single Center

Background: The benefits of intravenous tissue-type plasminogen activator (tPA) in acute ischemic stroke are time dependent, and national guidelines recommend door-to-needle (DTN) time within 60 minutes. Several strategies have been reported to be associated with reducing DTN times. However, effectiveness of such strategies has not been fully evaluated. Methods: In 2014, we assembled a multidisciplinary team called ‘Acute Stroke Team (AST)’ aiming for improving outcomes of patients with acute ischemic stroke, especially by reducing onset-to-treatment time. A new protocol was implemented to minimize delays: AST staff prenotification, parallel process workflow, and rapid acquisition of laboratory testing and brain imaging. AST reviewed all intravenous tPA cases and discussed the points of improvement. AST also organized both public and in-hospital lectures, and simulation training course. We compared patients received intravenous tPA within 4.5 hours from the symptom onset at our institute in the pre AST (April 2011 - August 2014) and post AST (September 2014 - July 2016) period. Using univariate methods and multivariable logistic regression, we assessed the associated factors with favorable outcomes. Results: In the pre and post AST period, 46 and 36 patients were treated with intravenous tPA, respectively. Compared with pre AST period, the median (interquartile range) DTN times was reduced from 71 (63-95) minutes to 55 (49-71) minutes (p&lt;0.01), and the percentage of patients with DTN times within 60 minutes were improved from 22% to 64% (P&lt;0.001) in the post AST period. By multivariable analysis, shorter DTN times (OR 0.98, 95% CI 0.95-0.99, p=0.025), lower age (OR 0.90, 95% CI 0.85-0.96, p=0.001) and lower NIHSS on admission (OR 0.88, 95% CI 0.82-0.95, p=0.001) were independently associated with independent ambulation at hospital discharge. Conclusions: Multidisciplinary team-based approach reduced DTN times. Reducing DTN times was associated with improving patient outcomes. Future efforts should focus on sustainability and safety of this approach.

Raw data queries during data-intensive parallel workflow execution

Computer simulations consume and produce huge amounts of raw data files presented in different formats, e.g., HDF5 in computational fluid dynamics simulations. Users often need to analyze domain-specific data based on related data elements from multiple files during the execution of computer simulations. In a raw data analysis, one should identify regions of interest in the data space and retrieve the content of specific related raw data files. Existing solutions, such as FastBit and RAW, are limited to a single raw data file analysis and can only be used after the execution of computer simulations. Scientific Workflow Management Systems (SWMS) can manage the dataflow of computer simulations and register related raw data files at a provenance database. This paper aims to combine the advantages of a dataflow-aware SWMS and the raw data file analysis techniques to allow for queries on raw data file elements that are related, but reside in separate files. We propose a component-based architecture, named as ARMFUL (Analysis of Raw data from Multiple Files) with raw data extraction and indexing techniques, which allows for a direct access to specific elements or regions of raw data space. ARMFUL innovates by using a SWMS provenance database to add a dataflow access path to raw data files. ARMFUL facilitates the invocation of ad-hoc programs and third party tools (e.g., FastBit tool) for raw data analyses. In our experiments, a real parallel computational fluid dynamics is executed, exploring different alternatives of raw data extraction, indexing and analysis.

The 14th International Symposium on Parallel and Distributed Computing

This special issue consists of six representative research articles presented at the 14th International Symposium on Parallel and Distributed Computing, in Limassol, a beautiful coastal Mediterranean city on the South coast of Cyprus. This annual symposium brings together practitioners, researchers, and scholars from the field of parallel and distributed computing to facilitate the exchange of ideas, enable collaborations, and promote the development of new research directions. The symposium had a highly selective program composed of papers describing original and unpublished research advancing the state of the art in the field of parallel and distributed computing. For this special issue, we selected the six best papers presented at the symposium. These papers reflect the broad nature of the field, addressing both theoretical and practical issues in mobile ad-hoc networks, clouds, routing, graphics processing unit computing, parallel irregular applications, security, and workflow scheduling. Alshareef and Grigoras 1 propose the use of a cloud service to register, save, pause, and resume sessions between mobile ad-hoc network (MANET) member nodes such that both the work in progress and energy are saved. They also introduce a checkpointing technique that captures the progress of a session and allows it to be resumed. Chilipirea et al.,2 describe a new fast simulator, designed to minimize the work needed to conduct extensive tests for opportunistic routing algorithms on multiple traces. They also present a comprehensive analysis of the most popular routing algorithms through extensive simulations conducted on their proposed simulation platform. Kouge et al.,3 propose a graphics processing unit implementation for digital halftoning employing local exhaustive search to produce high-quality binary images and to achieve significant speedup over the CPU implementation. Neves 4 presents two refinements of Exploit Parallelism in Irregular Codes (EPIC), a framework developed to ease the exploitation of task parallelism in irregular applications that use third-party tools and/or generate asymmetric sets of tasks. The two refinements focus on the software design and the scheduling algorithm of the EPIC framework. Sapegin et al.,5 present the design of a security information and event management system based on an in-memory database with an integrated machine learning library. Employing deep normalization of log messages, storing data in the main memory, and running data analysis directly in the database, the system achieves significant processing speeds allowing machine learning analysis of security events in almost real time. Xie et al.,6 propose three workflow scheduling algorithms, a fairness-based algorithm, a priority-based scheduling algorithm, and a tradeoff-based scheduling algorithm. The tradeoff-based algorithms were designed to meet the deadlines of more higher-priority workflows, while still allowing the lower-priority workflows to be processed actively for better performance. The guest editors would like to thank Prof. Geoffrey C. Fox for his help in putting together this special issue. The guest editors are very grateful to the reviewers for their high-quality reviews and constructive feedback on the papers.

ClowdFlows: Online workflows for distributed big data mining

The paper presents a platform for distributed computing, developed using the latest software technologies and computing paradigms to enable big data mining. The platform, called ClowdFlows, is implemented as a cloud-based web application with a graphical user interface which supports the construction and execution of data mining workflows, including web services used as workflow components. As a web application, the ClowdFlows platform poses no software requirements and can be used from any modern browser, including mobile devices. The constructed workflows can be declared either as private or public, which enables sharing the developed solutions, data and results on the web and in scientific publications. The server-side software of ClowdFlows can be multiplied and distributed to any number of computing nodes. From a developer’s perspective the platform is easy to extend and supports distributed development with packages. The paper focuses on big data processing in the batch and real-time processing mode. Big data analytics is provided through several algorithms, including novel ensemble techniques, implemented using the map-reduce paradigm and a special stream mining module for continuous parallel workflow execution. The batch mode and real-time processing mode are demonstrated with practical use cases. Performance analysis shows the benefit of using all available data for learning in distributed mode compared to using only subsets of data in non-distributed mode. The ability of ClowdFlows to handle big data sets and its nearly perfect linear speedup is demonstrated.

Scheduling Dynamic Parallel Loop Workflow in Cloud Environment

Scientific workflows have been employed to automate large scale scientific experiments by leveraging computational power provided on-demand by cloud computing platforms. Among these workflows, a parallel loop workflow is used for studying the effects of different input values of a scientific experiment. Because of its independent loop characteristic, a parallel loop workflow can be dynamically executed as parallel workflow instances to accelerate the execution. Such execution negates workflow traversal used in existing works to calculate execution time and cost during scheduling in order to maintain time and cost constraints. In this paper, we propose a novel scheduling technique that is able to handle dynamic parallel loop workflow execution through a new method for evaluating execution progress together with a workflow instance arrival control and a cloud resource adjustment mechanism. The proposed technique, which aims at maintaining a workflow deadline while reducing cost, is tested using 3 existing task scheduling heuristics as its task mapping strategies. The simulation results show that the proposed technique is practical and performs better when the time constraint is more relaxed. It also prefers task scheduling heuristics that allow for a more accurate progress evaluation.

PyCOMPSs: Parallel computational workflows in Python

The use of the Python programming language for scientific computing has been gaining momentum in the last years. The fact that it is compact and readable and its complete set of scientific libraries are two important characteristics that favour its adoption. Nevertheless, Python still lacks a solution for easily parallelizing generic scripts on distributed infrastructures, since the current alternatives mostly require the use of APIs for message passing or are restricted to embarrassingly parallel computations. In that sense, this paper presents PyCOMPSs, a framework that facilitates the development of parallel computational workflows in Python. In this approach, the user programs her script in a sequential fashion and decorates the functions to be run as asynchronous parallel tasks. A runtime system is in charge of exploiting the inherent concurrency of the script, detecting the data dependencies between tasks and spawning them to the available resources. Furthermore, we show how this programming model can be built on top of a Big Data storage architecture, where the data stored in the backend is abstracted and accessed from the application in the form of persistent objects.

A Dynamic Cloud Dimensioning Approach for Parallel Scientific Workflows: a Case Study in the Comparative Genomics Domain

Usually, scientists need to execute experiments that demand high performance computing environments and parallel techniques. This is the scenario found in many bioinformatics experiments modeled as scientific workflows, such as phylogenetic and phylogenomic analyses. To execute these experiments, scientists have adopted virtual machines (VMs) instantiated in clouds. Estimating the number of VMs to instantiate is a crucial task to avoid negative impacts on the execution performance and on the financial costs with under or overestimations. Previously, the necessary number of VMs to execute bioinformatics workflows have been estimated by a GRASP heuristic and have been coupled to a Cloud-based Parallel Scientific Workflow Management System. Although this work was a step forward, this approach only provided a static dimensioning. If the characteristics of the environment change (processing capacity, network speed), this static dimensioning may not be suitable. In this way, it is of interest that the dimensioning is adjusted at runtime. To achieve this, we developed a novel framework for monitoring and dynamically dimensioning resources during the execution of parallel scientific workflows in clouds, called Dynamic Dimensioning of Cloud Computing Framework (DDC-F). We have evaluated DDC-F in real executions of bioinformatics workflows. Experiments showed that DDC-F is able to efficiently calculate the number of VMs necessary to execute bioinformatics workflows of Comparative Genomics (CG), also reducing the financial costs, when compared with other works of the related literature.

PUMI

The Parallel Unstructured Mesh Infrastructure (PUMI) is designed to support the representation of, and operations on, unstructured meshes as needed for the execution of mesh-based simulations on massively parallel computers. In PUMI, the mesh representation is complete in the sense of being able to provide any adjacency of mesh entities of multiple topologies in O(1) time, and fully distributed to support relationships of mesh entities across multiple memory spaces in a manner consistent with supporting massively parallel simulation workflows. PUMI's mesh maintains links to the high-level model definition in terms of a model topology as produced by CAD systems, and is specifically designed to efficiently support evolving meshes as required for mesh generation and adaptation. To support the needs of parallel unstructured mesh simulations, PUMI also supports a specific set of services such as the migration of mesh entities between parts while maintaining the mesh adjacencies, maintaining read-only mesh entity copies from neighboring parts (ghosting), repartitioning parts as the mesh evolves, and dynamic mesh load balancing. Here we present the overall design, software structures, example programs, and performance results. The effectiveness of PUMI is demonstrated by its applications to massively parallel adaptive simulation workflows.