Parallel Dataflow Research Articles

A certain examination on heterogeneous systolic array (HSA) design for deep learning accelerations with low power computations

Acceleration techniques play a crucial role in enhancing the performance of modern high-speed computations, especially in Deep Learning (DL) applications where the speed is of utmost importance. One essential component in this context is the Systolic Array (SA), which effectively handles computational tasks and data processing in a rhythmic manner. Google's Tensor Processing Unit (TPU) leverages the power of SA for neural networks. The core SA's functionality and performance lies in the Computation Element (CE), which facilitates parallel data flow. In our article, we introduce a novel approach called Proposed Systolic Array (PSA), which is implemented on the CE and further enhanced with a modified Hybrid Kogge Stone adder (MHA). This design incorporates principles to expedite computations by rounding and extracting data model in SA as PSA-MHA. The PSA, utilizing a data flow model with MHA, significantly accelerates data shifts and control passes in execution cycles. We validated our approach through simulations on the Cadence Virtuoso platform using 65 nm process technology, comparing it to the General Matrix Multiplication (GMMN) benchmark. The results showed remarkable improvements in the CE, with a 30.29 % reduction in delay, a 23.07 % reduction in area, and an 11.87 % reduction in power consumption. The PSA outperformed these improvements, achieving a 46.38 % reduction in delay, a 7.58 % reduction in area, and an impressive 48.23 % decrease in Area Delay Product (ADP). To further substantiate our findings, we applied the PSA-based approach to pre-trained hybrid Convolutional and Recurrent (CNN-RNN) neural models. The PSA-based hybrid model incorporates 189 million Multiply-Accumulate (MAC) units, resulting in a weighted mean architecture value of 784.80 for the RNN component. We also explored variations in bit width, which led to delay reductions ranging from 20.17 % to 30.29 %, area variations between 13.08 % and 32.16 %, and power consumption changes spanning from 11.88 % to 20.42 %.

Features of Creating Parallel Programs for the Parallel Dataflow Computing System "Buran"

The use of modern multi-core and multi-processor computer systems for actual tasks is not effective enough, even taking into account the use of parallel programming technologies. The solution to the problem of efficient loading of computing resources can be the transition to computing models and architectures that are inherently parallel. One of such architectures is the Parallel Dataflow Computing System (PDCS) "Buran", which implements a dataflow computing model with a dynamically formed context. A feature of the dataflow computing model is the activation of computations by data readiness, which affects both the architecture of the computing system and the creation of programs for such systems. Differences between imperative and dataflow programming paradigms are also reflected in the route of creating a program, especially its parallel implementation. The route of creating a dataflow parallel program differs markedly from the traditional one. Already at the first stage, a parallel algorithm is created and implemented (including the algorithm for generating initial data). Next, this algorithm is debugged when it is executed on an emulator or model of the system with one computing core. After that, the selection and configuration of function of computation distribution is performed, the program is executed (without changing its code) on the emulator or model of the system with several computing cores, and, finally, the program is debugged in multi-core mode. These stages of the route differ from similar stages of the traditional route, both in form and in essence. Unlike traditional parallel, and even more so sequential programs, it can be said that a dataflow program in equal parts consists of a program code that implements a task, an algorithm for generating initial data, and a function of computation distribution. This thesis is demonstrated by the example of solving the problem of finding sum of array elements, where the difference between the implementations is in the algorithm for generating initial data, which radically affects the nature of the dataflow program passing. Careful attention to each of the parts of the dataflow program is the key to the correct and efficient solution of problems on the PDCS, which provides support to the programmer at the hardware level.

Transformation of Functional Dataflow Parallel Programs into Imperative Programs

Transformation of Functional Dataflow Parallel Programs into Imperative Programs

Aspects of Creating Parallel Programs in Dataflow Programming Paradigm

The imperative programming paradigm is the main one for creating sequential and parallel programs for the vast majority of modern computers, including supercomputers. A feature of the imperative paradigm is the sequence of commands. This feature is an obstacle to the creation of efficient parallel programs, since parallelism is achieved at the expense of additional code. One of the solutions to the problem of overhead for parallel computing is the creation of such a computing model and the architecture of the system that implements it, for which the parallel execution of the algorithm is an immanent property. This model is a dataflow computing model with a dynamically formed context and the architecture of the parallel dataflow computing system "Buran". A complete transition to dataflow systems is hampered, among other things, by the conceptual difference between the dataflow programming paradigm and the imperative one. The article compares these two paradigms. First, parallel data processing is an inherent property of a dataflow program. Second, the dataflow program consists of three elements: a set of initial data, a program code, and a parameterizable distribution function. And third, a conceptually different approach to the algorithmization of the task — the data themselves store information about who should process them (in traditional programs, on the contrary, the command stores information about what data should be processed). The article also presents the structure of a dataflow program and the route for creating a dataflow algorithm. The translation of basic algorithmic constructions (following, branching, loops) is considered on the example of simple problems.

HPVM: Hardware-Agnostic Programming for Heterogeneous Parallel Systems

We present Heterogeneous Parallel Virtual Machine (HPVM), a compiler framework for hardware-agnostic programming on heterogeneous compute platforms. HPVM introduces a hardware-agnostic parallel intermediate representation with constructs for the hierarchical task, data, and pipeline parallelism, including dataflow parallelism, and supports multiple front-end languages. In addition, HPVM provides optimization passes that navigate performance, energy, and accuracy tradeoffs, and includes retargetable back ends for a wide range of diverse hardware targets, including central processing units, graphics processing units, domain-specific accelerators, and field-programmable gate arrays. Across diverse hardware platforms, HPVM optimizations provide significant performance and energy improvements, while preserving object-code portability. With these capabilities, HPVM facilitates developers, domain experts, and hardware vendors in programming modern heterogeneous systems.

Systematically Understanding Graph Accelerator Dimensions and the Value of Hardware Flexibility

Because of the importance of graph workloads and the limitations of central processing units/graphics processing units (CPUs/GPUs), many graph-processing accelerators have been proposed. Most prior such accelerators adopt a single fixed algorithm. While helpful for specialization, this leaves performance potential from flexibility on the table and also complicates understanding the relationship between graph types, workloads, algorithms, and specialization. In this work, we explore the value of flexibility in graph-processing accelerators. Our approach is to identify a taxonomy of key algorithm variants, and develop a modular architecture, PolyGraph, which is flexible across them. The key to flexibility is our novel Taskflow execution model, which unifies task and dataflow parallelism. Overall, we find that flexibility is essential; PolyGraph outperforms similarly provisioned GPUs by mean 49.6× (up to 275×), and the best prior accelerator by mean 5.7×.

Scheduling Coflows With Dependency Graph

Applications in data-parallel computing typically consist of multiple stages. In each stage, a set of intermediate parallel data flows ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Coflow</i> ) is produced and transferred between servers to enable starting of next stage. While there has been much research on scheduling isolated coflows, the dependency between coflows in multi-stage jobs has been largely ignored. In this paper, we consider scheduling coflows of multi-stage jobs represented by general <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DAG</i> s (Directed Acyclic Graphs) in a shared data center network, so as to minimize the total weighted completion time of jobs. This problem is significantly more challenging than the traditional coflow scheduling, as scheduling even a single multi-stage job to minimize its completion time is shown to be NP-hard. In this paper, we propose a polynomial-time algorithm with approximation ratio of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$O(\mu \log (m)/\log (\log (m)))$ </tex-math></inline-formula> , where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mu $ </tex-math></inline-formula> is the maximum number of coflows in a job and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$m$ </tex-math></inline-formula> is the number of servers. For the special case that the jobs’ underlying dependency graphs are <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">rooted trees</i> , we modify the algorithm and improve its approximation ratio. To verify the performance of our algorithms, we present simulation results using real traffic traces that show up to 53% improvement over the prior approach. We conclude the paper by providing a result concerning an optimality gap for scheduling coflows with general DAGs.

Dynamic Control of Computation Consistency in the Parallel Dataflow Computing System

When developing high-performance multiprocessor computing systems, much attention is paid to ensuring uninterrupted operation, both in terms of hardware and software. In traditional computing systems, software is the main focus in address­ing these issues. The article discusses the solution to the issue of ensuring uninterrupted operation for the parallel dataflow computing system (PDCS), which implements the dataflow computational model with a dynamically formed context. Due to the features of the PDCS, it is proposed to implement this type of control in hardware, which will increase its efficiency, since the computational process will be controlled in dynamics, and not only in statics.

Improving The Performances of WSN Using Data Scheduler and Hierarchical Tree

Users of data-intensive implementation needs intelligent services and schedulers that will provide models and strategies to optimize their data transfer jobs. Normally sensor nodes are connected to consecutive sensor nodes depending on frequent transmission. To enhance end-to-end data flow parallelism for throughput optimization in high speed WSNs. The major objective is to maximize the WSNs throughput, minimizing the model overhead, avoiding disputation among users and using minimum number of end-system resources. Data packets are broadcasted from sender node to target node. Though, all nodes operate concurrently in various communications, the analysis shows that more packet latencies are occurred and priority-based transmission tasks are performed. Then the proposed Bearing parallelism-based Data Scheduler (BPDS) is used for data scheduling to enhance the end-to-end throughput input parameter. Sensor nodes are fast working node, it verifies each and every node before allocating packet transmission for that node. Busy resources are monitored to inform the nodes that are in processing, based on the schedule it allocates various paths to particular node and monitors the node capacity. Sampling algorithm supports for fixing threshold value, based on the values, they are further allocated to communicate between channels. It assigns the routing path with minimum resources and reduces end to end delay, to improve throughput, and network lifetime.

A Measurement-Based Message-Level Timing Prediction Approach for Data-Dependent SDFGs on Tile-Based Heterogeneous MPSoCs

Fast yet accurate performance and timing prediction of complex parallel data flow applications on multi-processor systems remains a very difficult discipline. The reason for it comes from the complexity of the data flow applications w.r.t. data dependent execution paths and the hardware platform with shared resources, like buses and memories. This combination may lead to complex timing interferences that are difficult to express in pure analytical or classical simulation-based approaches. In this work, we propose the combination of timing measurement and statistical simulation models for probabilistic timing and performance prediction of Synchronous Data Flow (SDF) applications on MPSoCs with shared memories. We exploit the separation of computation and communication in our SDF model of computation to set-up simulation-based performance prediction models following different abstraction approaches. We especially propose a message-level communication model driven by a data-dependent probabilistic execution phase timing model. We compare our work against measurement on two case-studies from the computer vision domain: a Sobel filter and a JPEG decoder. We show that the accuracy and execution time of our modeling and evaluation framework outperforms existing approaches and is suitable for a fast yet accurate design space exploration.

An Efficient and Flexible Accelerator Design for Sparse Convolutional Neural Networks

Designing hardware accelerators for convolutional neural networks (CNNs) has recently attracted tremendous attention. Plenty of existing accelerators are built for dense CNNs or structured sparse CNNs. By contrast, unstructured sparse CNNs can achieve higher compression ratio with equivalent accuracy. However, their corresponding hardware implementations generally suffer from load imbalance and conflict access to on-chip buffers, which results in under utilization of processing elements (PEs). To tackle these issues, we propose a hardware/power-efficient and highly flexible architecture to support both unstructured and structured sparse CNNs with various configurations. Firstly, we propose an efficient weight reordering algorithm to preprocess compressed weights and balance the workload of PEs. Secondly, an adaptive on-chip dataflow, namely hybrid parallel (HP) dataflow, is introduced to promote weight reuse. Thirdly, the partial fusion scheme, which was first introduced in one of our prior works, is incorporated as the off-chip dataflow. Benefited from dataflow optimizations, the repetitive data exchanges between on-chip buffers and external memories are significantly reduced. We implement the design on the Intel Arria10 SX660 platform and evaluate with MobileNet-v2, ResNet-50, and ResNet-18 on ImageNet dataset. Compared to existing sparse accelerators on FPGAs, the proposed accelerator can achieve $1.35\sim 1.81\times $ improvement in power efficiency with the same sparsity. Compared to prior dense accelerators, this accelerator can achieve an improvement of $1.92\sim 5.84\times $ in DSP efficiency.

The System for Transforming the Code of Dataflow Programs into Imperative

Functional dataflow programming languages are designed to create parallel portable programs. The source code of such programs is translated into a set of graphs that reflect information and control dependencies. The main way of their execution is interpretation, which does not allow to perform calculations efficiently on real parallel computing systems and leads to poor performance. To run programs directly on existing computing systems, you need to use specific optimization and transformation methods that take into account the features of both the programming language and the architecture of the system. Currently, the most common is the Von Neumann architecture, however, parallel programming for it in most cases is carried out using imperative languages with a static type system. For different architectures of parallel computing systems, there are various approaches to writing parallel programs. The transformation of dataflow parallel programs into imperative programs allows to form a framework of imperative code fragments that directly display sequential calculations. In the future, this framework can be adapted to a specific parallel architecture. The paper considers an approach to performing this type of transformation, which consists in allocating fragments of dataflow parallel programs as templates, which are subsequently replaced by equivalent fragments of imperative languages. The proposed transformation methods allow generating program code, to which various optimizing transformations can be applied in the future, including parallelization taking into account the target architecture.

Daisy: Data analysis integrated software system for X-ray experiments

Daisy (Data Analysis Integrated Software System) has been designed for the analysis and visualisation of X-ray experiments. To address the requirements of the Chinese radiation facilities community, spanning an extensive range from purely algorithmic problems to scientific computing infrastructure, Daisy sets up a cloud-native platform to support on-site data analysis services with fast feedback and interaction. Furthermore, the plug-in based application is convenient to process the expected high throughput data flow in parallel at next-generation facilities such as the High Energy Photon Source (HEPS). The objectives, functionality and architecture of Daisy are described in this article.

OctCNN: A High Throughput FPGA Accelerator for CNNs using Octave Convolution Algorithm

With the rapid development of convolutional neural networks (CNNs), FPGAs have become one of the most attractive candidates for deploying CNNs. However, previous FPGA solutions based on the traditional convolution are still limited by computational power. In this article, we introduce the octave convolution (OctConv) into the CNN accelerator design for the first time to improve the hardware acceleration efficiency and design a dedicated OctPU for mapping OctConv to FPGAs, which employs a parallel dataflow pattern to exploit the parallelism of OctConv. Then, we present a novel and scalable architecture that dynamically combines the inter-layer pipelined structure and multi-layer reuse structure. Meanwhile, to obtain the optimized solution, we build a multidimensional performance and resource analysis model and a two-stage search algorithm based on greedy and heuristic algorithms. We evaluate our proposal by implementing VGG16 and ResNet50 on the Xilinx VU9P FPGA. Experimental results show that our prototypes can achieve an average of 3321 GOP/s for the convolutional layers for VGG16 and 2873 GOP/s for the overall ResNet50 using OctConv. Compared to previous works based on the traditional convolution, our prototypes own a 1.72 to 2.33 speedup in throughput and a 2.01 to 5.18 improvement in computational density. Our design also presents an excellent compromise performance and generalization

Concurrency Analysis in Dynamic Dataflow Graphs

Dynamic dataflow scheduling enables effective exploitation of concurrency while making parallel programming easier. To this end, analyzing the inherent degree of concurrency available in dataflow graphs is an important task, since it may aid compilers or programmers to assess the potential performance a program can achieve via parallel execution. However, traditional concurrency analysis techniques only work for DAGs (directed acyclic graphs), hence the need for new techniques that contemplate graphs with cycles. In this paper we present techniques to perform concurrency analysis on generic dynamic dataflow graphs, even in the presence of cycles. In a dataflow graph, nodes represent instructions and edges describe dependencies. The novelty of our approach is that we allow concurrency between different iterations of the loops. Consequently, a set of concurrent nodes may contain instructions from different loops that can be proven independent. In this work, we provide a set of theoretical tools for obtaining bounds and illustrate implementation of parallel dataflow runtime on a set of representative graphs for important classes of benchmarks to compare measured performance against derived bounds.

A Ubiquitous Machine Learning Accelerator With Automatic Parallelization on FPGA

Machine learning has been widely applied in various emerging data-intensive applications, and has to be optimized and accelerated by powerful engines to process very large scale data. Recently, the instruction set based accelerators on Field Progarmmable Gate Arrays (FPGAs) have been a promising topic for machine learning applications. The customized instructions can be further scheduled to achieve higher instruction-level parallelism. In this article, we design a ubiquitous accelerator with out-of-order automatic parallelization for large-scale data-intensive applications. The accelerator accommodates four representative applications, including clustering algorithms, deep neural networks, genome sequencing, and collaborative filtering. In order to improve the coarse-grained instruction-level parallelism, the accelerator employs an out-of-order scheduling method to enable parallel dataflow computation. We use Colored Petri Net (CPN) tools to analyze the dependences in the applications, and build a hardware prototype on the real FPGA platform. For cluster applications, the accelerator can support four different algorithms, including K-Means, SLINK, PAM, and DBSCAN. For collaborative filtering applications, it accommodates Tanimoto, euclidean, Cosine, and Pearson Correlation as Similarity metrics. For deep learning applications, we implement hardware accelerators for both training process and inference process. Finally, for genome sequencing, we design a hardware accelerator for the BWA-SW algorithm. Experimental results show that the accelerator architecture can reach up to 25X speedup against Intel processors with affordable hardware cost, insignificant power consumption, and high flexibility.

NZESPA: A Near-3D-Memory Zero Skipping Parallel Accelerator for CNNs

Convolutional neural networks (CNNs) are one of the most popular machine learning tools for computer vision. The ubiquitous use in several applications with its high computation-cost has made it lucrative for optimization through accelerated architecture. State-of-the-art has either exploited the parallelism of CNNs, or eliminated computations through sparsity or used near-memory processing (NMP) to accelerate the CNNs. We introduce NMP-fully sparse architecture, which acquires all three capabilities. The proposed architecture is parallel and hence processes the independent CNN tasks concurrently. To exploit the sparsity, the proposed system employs a dataflow, namely, Near-3D-Memory Zero Skipping Parallel dataflow or nZESPA dataflow. This dataflow maintains the compressed-sparse encoding of data that skips all ineffectual zero-valued computations of CNNs. We design a custom accelerator which employs the nZESPA dataflow. The grids of nZESPA modules are integrated into the logic layer of the hybrid memory cube. This integration saves a significant amount of off-chip communications while implementing the concept of NMP. We compare the proposed architecture with three other architectures which either do not exploit sparsity (NMP-dense) or do not employ NMP (traditional-fully sparse) or do not include both (traditional-dense). The proposed system outperforms the baselines in terms of performance and energy consumption while executing CNN inference.

Parallel Designs for Metaheuristics that Solve Portfolio Selection Problems Using Fuzzy Outranking Relations

In decision-making, the multiobjective portfolio selection problem (MPSP) consists of the selection of alternatives based on preferences of a particular decision-maker (DM). In real-world applications, MPSP has several conflicting criteria that DMs must consider to determine an appropriate solution. So far, only fuzzy outranking relations have been used in a relational system of preferences to guide the search process of genetic algorithms (NOSGAII), and ant colony optimization (NOACO) to approximate the region of interest (RoI) of MPSP involving DM’s preferences. The NOSGAII and NOACO strategies are sequential, and they face a real challenge when solving high-dimensional instances which is a decrement in the computational efficiency due to the increment in the number of objectives. The present research proposes to use parallelism to tackle the efficiency situation in metaheuristics. The study first identifies which approach approximates better the RoI, and then, it analyzes the effect of parallelism in the performance. The results showed that NOACO found more best compromises in almost all the instances than NOSGAII. Hence, it can be concluded that NOACO approximates better the RoI. Also, the results showed a better average speedup with coarse-grained parallelism in NOACO than with data-flow parallelism, suggesting the conclusion that ants independently working are faster than ants working collaboratively. Finally, the main contributions are (1) the analysis of the performance of the two approaches for MPSP, (2) the five parallel designs for NOACO, and (3) the parallel NOACO that speedups up to 2× the sequential version when solving MPSP.

Методы и подходы к повышению надежности параллельной потоковой вычислительной системы

Методы и подходы к повышению надежности параллельной потоковой вычислительной системы

Влияние особенностей модели вычислений и архитектуры на надежность параллельной потоковой вычислительной системы

Влияние особенностей модели вычислений и архитектуры на надежность параллельной потоковой вычислительной системы