Dynamic Pipelines Research Articles

Pharmosome: an integrative and collective database for exploration and analysis of single nucleotide polymorphisms associated with disease

Current single nucleotide polymorphism (SNP) databases are limited to a narrow set of SNPs, which has led to a lack of interactivity between different databases, limited tools to analyze and manipulate the already existing data, and complexity in the graphical user interface. Here we introduce Pharmosome, a web-based, user-friendly and collective database for more than 30,000 human disease-related SNPs, with dynamic pipelines to explore SNPs associated with disease development, drug response and the pathways shared between different genes related to these SNPs. Pharmosome implements several tools to design primers to detect SNPs in large genomes and facilitates analysis of different SNPs to determine relationships between them by aligning sequences, constructing phylogenetic trees, and providing consensus sequences illustrating the connections between SNPs. Pharmosome was written in the Python programming language using the Django web framework in combination with HTML, CSS, and JavaScript to receive user inputs, and process and export the sorted result to the interface. Pharmosome is available from: https://pharmosome.herokuapp.com/.

An optimization framework for dynamic pipeline management in computing systems

Dynamic computer pipelines use feedback loops for iterative computation, due to which, data items, while traversing them often collide with one another. These pipelines, therefore, employ specialized mechanisms to detect and avoid such collisions in real-time. Mostly, this is done by deciding when, and when not, to allow newer data items to enter the pipeline. We will show that such decisions may lead to different throughput, and guaranteeing optimal performance is an optimization problem. We begin by building a mathematical model of dynamic pipelines, and make use of genetic algorithm to yield near-optimal throughput. The proposed optimization technique accounts for the hardware overhead, incurred due to the extensions in the pipeline organization, as suggested by itself. Our confidence in the results stems from simulation of 10,000 dynamic pipeline organizations, and for verification of results, we present hardware implementation of one of those. The proposed framework will specifically be useful in embedded systems for digital signal processing applications.

SciPipe: A workflow library for agile development of complex and dynamic bioinformatics pipelines.

BackgroundThe complex nature of biological data has driven the development of specialized software tools. Scientific workflow management systems simplify the assembly of such tools into pipelines, assist with job automation, and aid reproducibility of analyses. Many contemporary workflow tools are specialized or not designed for highly complex workflows, such as with nested loops, dynamic scheduling, and parametrization, which is common in, e.g., machine learning.FindingsSciPipe is a workflow programming library implemented in the programming language Go, for managing complex and dynamic pipelines in bioinformatics, cheminformatics, and other fields. SciPipe helps in particular with workflow constructs common in machine learning, such as extensive branching, parameter sweeps, and dynamic scheduling and parametrization of downstream tasks. SciPipe builds on flow-based programming principles to support agile development of workflows based on a library of self-contained, reusable components. It supports running subsets of workflows for improved iterative development and provides a data-centric audit logging feature that saves a full audit trace for every output file of a workflow, which can be converted to other formats such as HTML, TeX, and PDF on demand. The utility of SciPipe is demonstrated with a machine learning pipeline, a genomics, and a transcriptomics pipeline.ConclusionsSciPipe provides a solution for agile development of complex and dynamic pipelines, especially in machine learning, through a flexible application programming interface suitable for scientists used to programming or scripting.

Comparing MapReduce and Pipeline Implementations for Counting Triangles

A common method to define a parallel solution for a computational problem consists in finding a way to use the Divide and Conquer paradigm in order to have processors acting on its own data and scheduled in a parallel fashion. MapReduce is a programming model that follows this paradigm, and allows for the definition of efficient solutions by both decomposing a problem into steps on subsets of the input data and combining the results of each step to produce final results. Albeit used for the implementation of a wide variety of computational problems, MapReduce performance can be negatively affected whenever the replication factor grows or the size of the input is larger than the resources available at each processor. In this paper we show an alternative approach to implement the Divide and Conquer paradigm, named dynamic pipeline. The main features of dynamic pipelines are illustrated on a parallel implementation of the well-known problem of counting triangles in a graph. This problem is especially interesting either when the input graph does not fit in memory or is dynamically generated. To evaluate the properties of pipeline, a dynamic pipeline of processes and an ad-hoc version of MapReduce are implemented in the language Go, exploiting its ability to deal with channels and spawned processes. An empirical evaluation is conducted on graphs of different topologies, sizes, and densities. Observed results suggest that dynamic pipelines allows for an efficient implementation of the problem of counting triangles in a graph, particularly, in dense and large graphs, drastically reducing the execution time with respect to the MapReduce implementation.

The Design of High-Performance Dynamic Asynchronous Pipelines: High-Capacity Style

This paper introduces a high-throughput asynchronous pipeline style, called high-capacity (HC) pipelines, targeted to datapaths that use dynamic logic. This approach includes a novel highly-concurrent handshake protocol, with fewer synchronization points between neighboring pipeline stages than almost all existing asynchronous dynamic pipelining approaches. Furthermore, the dynamic pipelines provide 100% buffering capacity, without explicit latches, by means of separate pullup and pulldown control for each pipeline stage: neighboring stages can store distinct data items, unlike almost all existing latchless dynamic asynchronous pipelines. As a result, very high throughput is obtained. Fabricated first-input-first-output (FIFO) designs, in 0.18-m technology, were fully functional over a wide range of supply voltages (1.2 to over 2.5 V), exhibiting a corresponding range of throughputs from 1.0-2.4 giga items/s. In addition, an experimental finite-impulse response (FIR) filter chip was designed and fabricated with IBM Research, whose speed-critical core used an HC pipeline. The HC pipeline exhibited throughputs up to 1.8 giga items/s, and the overall filter achieved 1.32 giga items/s, thus obtaining 15% higher throughput and 50% lower latency than the fastest previously-reported synchronous FIR filter, also designed at IBM Research.

Dynamic computational blocks for bit-level systolic arrays

Integrated dynamic logic trees with latches provide cost effective circuit techniques for building massively pipelined, systolic, computational blocks operating at the bit level. Recent results have demonstrated that dynamic pipelines are capable of very high switching speeds with appropriate circuit design techniques. In this paper we trade some of this speed for much higher functionality of each logic block. The resulting throughput rate remains sufficiently high for useful applications, but results in substantial area and power savings. Design techniques for the individual logic trees (switching trees) are based on simple graph theoretic rules. Examples are shown to support the technique.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>