Dataflow Architecture Research Articles

HLPerf: Demystifying the Performance of HLS-based Graph Neural Networks with Dataflow Architectures

The development of FPGA-based applications using HLS is fraught with performance pitfalls and large design space exploration times. These issues are exacerbated when the application is complicated and its performance is dependent on the input dataset, as is often the case with graph neural network approaches to machine learning. Here, we introduce HLPerf, an open-source, simulation-based performance evaluation framework for dataflow architectures that both supports early exploration of the design space and shortens the performance evaluation cycle. We apply the methodology to GNNHLS, an HLS-based graph neural network benchmark containing six commonly used graph neural network models and four datasets with distinct topologies and scales. The results show that HLPerf achieves over 10, 000× average simulation acceleration relative to RTL simulation and over 400× acceleration relative to state-of-the-art cycle-accurate tools at the cost of 7% mean error rate relative to actual FPGA implementation performance. This acceleration positions HLPerf as a viable component in the design cycle.

Understanding the Potential of FPGA-based Spatial Acceleration for Large Language Model Inference

Recent advancements in large language models (LLMs) boasting billions of parameters have generated a significant demand for efficient deployment in inference workloads. While hardware accelerators for Transformer-based models have been extensively studied, the majority of existing approaches rely on temporal architectures that reuse hardware units for different network layers and operators. However, these methods often encounter challenges in achieving low latency due to considerable memory access overhead. This article investigates the feasibility and potential of model-specific spatial acceleration for LLM inference on field-programmable gate arrays (FPGAs). Our approach involves the specialization of distinct hardware units for specific operators or layers, facilitating direct communication between them through a dataflow architecture while minimizing off-chip memory accesses. We introduce a comprehensive analytical model for estimating the performance of a spatial LLM accelerator, taking into account the on-chip compute and memory resources available on an FPGA. This model can be extended to multi-FPGA settings for distributed inference. Through our analysis, we can identify the most effective parallelization and buffering schemes for the accelerator and, crucially, determine the scenarios in which FPGA-based spatial acceleration can outperform its GPU-based counterpart. To enable more productive implementations of an LLM model on FPGAs, we further provide a library of high-level synthesis (HLS) kernels that are composable and reusable. This library will be made available as open-source. To validate the effectiveness of both our analytical model and HLS library, we have implemented Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformers (GPT2) on an AMD Xilinx Alveo U280 FPGA device. Experimental results demonstrate our approach can achieve up to 13.4× speedup when compared to previous FPGA-based accelerators for the BERT model. For GPT generative inference, we attain a 2.2× speedup compared to Design for Excellence, an FPGA overlay, in the prefill stage, while achieving a 1.9× speedup and a 5.7× improvement in energy efficiency compared to the NVIDIA A100 GPU in the decode stage.

Reconfigurable Acceleration of Neural Networks: A Comprehensive Study of FPGA-based Systems

This paper explores the potential of Field-Programmable Gate Arrays (FPGAs) for accelerating both neural network inference and training. We present a comprehensive analysis of FPGA-based systems, encompassing architecture design, hardware implementation strategies, and performance evaluation. Our study highlights the advantages of FPGAs over traditional CPUs and GPUs for neural network workloads, including their inherent parallelism, reconfigurability, and ability to tailor hardware to specific network needs. We delve into various hardware implementation strategies, from direct mapping to dataflow architectures and specialized hardware blocks, examining their impact on performance. Furthermore, we benchmark FPGA-based systems against traditional platforms, evaluating inference speed, energy efficiency, and memory bandwidth. Finally, we explore emerging trends in FPGA-based neural network acceleration, such as specialized architectures, efficient memory management techniques, and hybrid CPU-FPGA systems. Our analysis underscores the significant potential of FPGAs for accelerating deep learning applications, particularly those requiring high performance, low latency, and energy efficiency.

Domain Specific Abstractions for the Development of Fast-by-Construction Dataflow Codes on FPGAs

FPGAs are popular in many fields but have yet to gain wide acceptance for accelerating HPC codes. A major cause is that whilst the growth of High-Level Synthesis (HLS), enabling the use of C or C++, has increased accessibility, without widespread algorithmic changes these tools only provide correct-by-construction rather than fast-by-construction programming. The fundamental issue is that HLS presents a Von Neumann-based execution model that is poorly suited to FPGAs, resulting in a significant disconnect between HLS’s language semantics and how experienced FPGA programmers structure dataflow algorithms to exploit hardware. We have developed the high-level language Lucent which builds on principles previously developed for programming general-purpose dataflow architectures. Using Lucent as a vehicle, in this paper we explore appropriate abstractions for developing application-specific dataflow machines on reconfigurable architectures. The result is an approach enabling fast-by-construction programming for FPGAs, delivering competitive performance against hand-optimised HLS codes whilst significantly enhancing programmer productivity.

ReSA: Reconfigurable Systolic Array for Multiple Tiny DNN Tensors

Systolic array architecture has significantly accelerated deep neural networks (DNNs). A systolic array comprises multiple processing elements (PEs) that can perform multiply-accumulate (MAC). Traditionally, the systolic array can execute a certain amount of tensor data that matches the size of the systolic array simultaneously at each cycle. However, hyper-parameters of DNN models differ across each layer and result in various tensor sizes in each layer. Mapping these irregular tensors to the systolic array while fully utilizing the entire PEs in a systolic array is challenging. Furthermore, modern DNN systolic accelerators typically employ a single dataflow. However, such a dataflow isn’t optimal for every DNN model. This work proposes ReSA, a reconfigurable dataflow architecture that aims to minimize the execution time of a DNN model by mapping tiny tensors on the spatially partitioned systolic array. Unlike conventional systolic array architectures, the ReSA data path controller enables the execution of the input, weight, and output-stationary dataflow on PEs. ReSA also decomposes the coarse-grain systolic array into multiple small ones to reduce the fragmentation issue on the tensor mapping. Each small systolic sub-array unit relies on our data arbiter to dispatch tensors to each other through the simple interconnected network. Furthermore, ReSA reorders the memory access to overlap the memory load and execution stages to hide the memory latency when tackling tiny tensors. Finally, ReSA splits tensors of each layer into multiple small ones and searches for the best dataflow for each tensor on the host side. Then, ReSA encodes the predefined dataflow in our proposed instruction to notify the systolic array to switch the dataflow correctly. As a result, our optimization on the systolic array architecture achieves a geometric mean speedup of 1.87X over the weight-stationary systolic array architecture across 9 different DNN models.

CUTE: A scalable CPU-centric and Ultra-utilized Tensor Engine for convolutions

Convolution is a fundamental and computationally expensive primitive and finds ubiquitous in deep neural networks (DNNs). The evolving DNNs have spurred the emergence of numerous accelerators and they successfully achieve high throughput. However, for DNN inference with small batch sizes, the computational resources of the accelerators are often under-utilized, and the overhead of offloading is significant. Compared to accelerators, the CPU can better meet fast response requirements of inference, flexibly handle various models, and is suitable for various scenarios (from edge to data center). Therefore, CPU remains an attractive platform for DNN inference, despite the sub-optimal performance, and resource efficiency.In this paper, we propose CUTE, a scalable CPU-centric and ultra-utilized tensor engine for convolutions. It co-designs data flow and hardware architecture to leverage the data reuse and parallelism of convolutions. CUTE is composed of several small tensor elements (TEs) and two-level buffers. It employs a decoupled access-execution architecture and greedy strategy to feed data to TEs, enabling it to achieve ultra utilization and great scalability. CUTE is tightly coupled with the CPU to minimize offloading latency, thereby providing efficient convolution computing capabilities for the system. Experimental results show that under the same bandwidth, CUTE achieves an average performance improvement of 3.8x compared with the CPU AVX512 unit and 1.6x compared with the CPU AMX unit. Besides, CUTE achieves a speedup of 7.0x and 3.9x over Nvidia V100 GPU and Eyeriss accelerator respectively, due to higher utilization of computing units.

Improving Utilization of Dataflow Unit for Multi-Batch Processing

Dataflow architectures can achieve much better performance and higher efficiency than general-purpose core, approaching the performance of a specialized design while retaining programmability. However, advanced application scenarios place higher demands on the hardware in terms of cross-domain and multi-batch processing. In this article, we propose a unified scale-vector architecture that can work in multiple modes and adapt to diverse algorithms and requirements efficiently. First, a novel reconfigurable interconnection structure is proposed, which can organize execution units into different cluster typologies as a way to accommodate different data-level parallelism. Second, we decouple threads within each DFG node into consecutive pipeline stages and provide architectural support. By time-multiplexing during these stages, dataflow hardware can achieve much higher utilization and performance. In addition, the task-based program model can also exploit multi-level parallelism and deploy applications efficiently. Evaluated in a wide range of benchmarks, including digital signal processing algorithms, CNNs, and scientific computing algorithms, our design attains up to 11.95× energy efficiency (performance-per-watt) improvement over GPU (V100), and 2.01× energy efficiency improvement over state-of-the-art dataflow architectures.

Coupled Ferroelectric-Photonic Memory in a Retinomorphic Hardware for In-Sensor Computing.

The development of all-in-one devices for artificial visual systems offers an attractive solution in terms of energy efficiency and real-time processing speed. In recent years, the proliferation of smart sensors in the growth of Internet-of-Things (IoT) has led to the increasing importance of in-sensor computing technology, which places computational power at the edge of the data-flow architecture. In this study, a prototype visual sensor inspired by the human retina is proposed, which integrates ferroelectricity and photosensitivity in two-dimensional (2D) α-In2Se3 material. This device mimics the functions of photoreceptors and amacrine cells in the retina, performing optical reception and memory computation functions through the use of electrical switching polarization in the channel. The gate-tunable linearity of excitatory and inhibitory functions in photon-induced short-term plasticity enables to encode and classify 12000 images in the Mixed National Institute of Standards and Technology (MNIST) dataset with remarkable accuracy, achieving ≈94%. Additionally, in-sensor convolution image processing through a network of phototransistors, with five convolutional kernels electrically pre-programmed into the transistors is demonstrated. The convoluted photocurrent matrices undergo straightforward arithmetic calculations to produce edge and feature-enhanced scenarios. The findings demonstrate the potential of ferroelectric α-In2Se3 for highly compact and efficient retinomorphic hardware implementation, regardless of ambipolar transport in the channel.

Unleashing the power of decentralized serverless IoT dataflow architecture for the Cloud-to-Edge Continuum: a performance comparison

The advent of new computing and communication trends that link pervasive data sources and consumers, such as Edge Computing, 5G and IIoT, has led to the development of the Cloud-to-Edge Continuum in order to take advantage of the resources available in massive IoT scenarios and to conduct data analysis to leverage intelligence at all levels. This paper outlines the challenging requirements of this novel IoT context and presents an innovative IoT framework to develop dataflow applications for data-centric environments. The proposed design takes advantage of decentralized Pub/Sub communication and serverless nanoservice architecture, using novel technologies such as Zenoh and WebAssembly, respectively, to implement lightweight services along the Cloud-to-Edge infrastructure. We also describe some use cases to illustrate the benefits and concerns of the coming IoT generation, giving a communication performance comparison of Zenoh over brokered MQTT strategies.Graphical abstract

Merging control-flow and dataflow architectures on a single chip

Computing power rises predominantly by increasing the number of cores in modern processors and the number of processors in cluster and cloud architectures. Along with increasing processing power, high-performance computing requirements also rise. The majority of the computing infrastructure includes control-flow processors that are based on the von Neumann paradigm. On the contrary, the principle of dataflow architectures is based on the data flowing through the already configured hardware. Recent research has proposed hybrid architectures, where both control-flow and dataflow hardware would exist on the same chip die. This article proposes a new hybrid control-flow and dataflow architecture where the control-flow hardware resembles modern graphical cards with thousands of cores and each GPU core has a reasonable amount of data-flow hardware. In this way, the advantages of dataflow architecture are exploited, including faster processing of high-performance computing algorithms and lower power consumption, while the conventional problem of communicating between control-flow and dataflow architectures is minimized. The proposed architecture is tested by analyzing the conjugate gradient method executed on both control-flow and dataflow hardware. The execution of the algorithm is divided onto GPU cores, and the execution of repeated instructions on each GPU core is delegated to the assigned dataflow hardware. The results indicate that it is possible to accelerate the execution of algorithms using the proposed architecture.

A TinyML Model for Gesture-Based Air Handwriting Arabic Numbers Recognition

In an era where the demand for efficient and practical machine learning (ML) solutions on resource-constrained devices is evergrowing, the realm of tiny machine learning (TinyML) emerges as a promising frontier. Motivated by the need for lightweight, low-power models that can be deployed on edge devices, this research paper presents an innovative TinyML model tailored to recognize Arabic hand gestures executed in mid-air. With a primary emphasis on the precise classification of Arabic numbers through these expressive hand movements, the paper unveils a comprehensive dataflow architecture. This intricate architecture processes accelerometer and gyroscope data to derive exact 2D gesture coordinates, a fundamental component of the recognition process. The cornerstone of the proposed model is the integration of Convolutional Neural Networks (CNNs), elucidating their exceptional role in achieving an impressive 93.8% accuracy rate in the classification of diverse Arabic Numbers gestures. This remarkable level of precision underscores the model's efficacy and resilience, rendering it an ideal candidate for real-time deployment in various gesture recognition scenarios.

Time, causality, and realizability: Engineering interactive, distributed software systems

Today, software systems are distributed, concurrent, interactive, embedded, and often time-critical. They communicate and cooperate by data exchange. Introducing notions of time, causality, and realizability into interface-based data flow models results into a powerful model for engineering cyber-physical and, generally, distributed software systems. Architectures and their specification are designed by the concurrent composition of systems. This approach provides high expressive power supporting explicitly specification of interface behavior, encapsulation, abstraction, refinement, verification, concurrent composition, and modularity. The key idea is a formal, semantically coherent modelling, and specification framework that integrates the specification of time-critical and non-time-critical software embedded into cyber-physical systems. The approach supports the refinement of interface specifications of non-time-critical systems into interface specifications of time-critical systems and vice versa. This leads to an integrated specification and design approach for time-critical data flow architectures with soft, firm as well as hard time-critical requirements.

On the RTL Implementation of FINN Matrix Vector Unit

Field-programmable gate array (FPGA)–based accelerators are becoming increasingly popular for deep neural network (DNN) inference due to their ability to scale performance with increasing degrees of specialization with dataflow architectures or custom data type precision. In order to reduce the barrier for software engineers and data scientists to adopt FPGAs, C++- and OpenCL-based design entries with high-level synthesis (HLS) have been introduced. They provide higher abstraction compared with register-transfer level (RTL)–based design. HLS offers faster development time, better maintainability, and more flexibility in code exploration when evaluating several options for multi-dimension tensors, convolutional layers, or different degrees of parallelism. For this reason, HLS has been adopted by DNN accelerator generation frameworks such as FINN and hls4ml. In this article, we present an alternative backend library for FINN, leveraging RTL. We investigate and evaluate, across a spectrum of design dimensions, the pros and cons of an RTL-based implementation versus the original HLS variant. We show that for smaller design parameters, RTL produces significantly smaller circuits as compared with HLS. For larger circuits, however, the look-up table (LUT) count of RTL-based design is slightly higher, up to around 15%. On the other hand, HLS consistently requires more flip-flops (FFs; with an orders-of-magnitude difference for smaller designs) and block RAMs (BRAMs; 2× more). This also impacts the critical path delay, with RTL producing significantly faster circuits, up to around 80%. RTL also benefits from at least a 10× reduction in synthesis time. Finally, the results were validated in practice using two real-world use cases, one of a multi-layer perceptron (MLP) used in network intrusion detection and the other a convolution network called ResNet, used in image recognition. Overall, since HLS frameworks code-generate the hardware design, the benefits of the ease in the design entry is less important. As such, the gained benefits in synthesis time together with some design-dependent resource benefits make the RTL abstraction an attractive alternative.

An area-efficient and low-latency elliptic curve scalar multiplication accelerator over prime field

For the elliptic curve cryptography (ECC) over prime field, this paper presents an area-efficient and low-latency elliptic curve scalar multiplication (ECSM) accelerator, which supports to process any prime number p ≤ 256 bits. The proposed accelerator is based on a parallel hardware implementation of iterative digit-digit Montgomery multiplication and an optimized data flow architecture for elliptic curve point operations. The proposed design is implemented in Xilinx Virtex-7 FPGA. The experimental results show that the proposed architecture occupies 23.7k look-up tables (LUTs) and performs single 256-bit scalar multiplication in 0.56 ms. The area-time product of the proposed architecture is only 13.36. Compared with other state-of-the-art ECSM implementations, the proposed architecture has obvious advantage in terms of the area-time product.

HMT: A Hardware-centric Hybrid Bonsai Merkle Tree Algorithm for High-performance Authentication

The Bonsai Merkle tree (BMT) is a widely used tree structure for authentication of metadata such as encryption counters in a secure computing system. Common BMT algorithms were designed for traditional Von Neumann architectures with a software-centric implementation in mind and as such, they are predominantly recursive and sequential in nature. However, the modern heterogeneous computing platforms employing Field-Programmable Gate Array (FPGA) devices require concurrency-focused algorithms to fully utilize the versatility and parallel nature of such systems. The recursive nature of traditional BMT algorithms makes them challenging to implement in such hardware-based setups. Our goal for this work is to introduce HMT, a hardware-friendly BMT algorithm that enables the verification and update processes to function independently and provides the benefits of relaxed update while being comparable to the eager update in terms of update complexity. The methodology of HMT contributes both novel algorithmic revisions and innovative hardware techniques to implementing BMT. We mathematically demonstrate the challenges of potentially unbounded recursions in relaxed BMT updates. To solve this problem, we use a partitioned BMT caching scheme that allocates a separate write-back cache for each BMT level—thus allowing for low and fixed upper bounds for dirty evictions compared to the traditional BMT caches. Then we introduce the aforementioned hybrid BMT algorithm that is hardware-targeted, parallel, and relaxes the update depending on BMT cache hit but makes the update conditions more flexible compared to lazy update to save additional write-backs. Deploying this new algorithm, we have designed a new BMT controller with a dataflow architecture including speculative buffers and parallel write-back engines to facilitate performance-enhancing mechanisms (like multiple concurrent authentication and independent updates) that were not possible with the conventional lazy algorithm. Our empirical performance measurements on a Xilinx U200 accelerator FPGA have demonstrated that HMT can achieve up to 7× improvement in bandwidth and 4.5× reduction in latency over lazy-update BMT baseline and up to 14% faster execution in standard benchmarks compared to a state-of-the-art, eager-update BMT solution.

PicoTDC: a flexible 64 channel TDC with picosecond resolution

The PicoTDC is a 64 channel TDC (Time to Digital Converter) ASIC, with 3ps or 12ps time binning, developed at CERN for use in a large variety of high channel count scientific instrumentation. Acquired time measurements are processed on-chip with a programmable buffering and triggered data flow architecture before being read out on 1 to 4 byte-wise readout ports with a total readout bandwidth of up to 10 Gbps. Leading edge, falling edge or leading edge together with pulse width can be captured at burst rates as high as 1.2 GHz and sustained rate of 320 MHz per channel. Effective single-shot time resolution has been measured to be 3.7 ps RMS across all 64 channels over its full dynamic range and as good as 1.35 ps RMS when using an on-chip adjustment feature optimized for a specific channel (0.43 ps if averaged over multiple measurements). The PicoTDC is implemented in a 65 nm CMOS process and 20 k chips have been produced and packaged in a 400 pin plastic BGA package.

A High-Throughput Full-Dataflow MobileNetv2 Accelerator on Edge FPGA

FPGA accelerators for lightweight neural networks such as MobileNetv2 are of great need in edge computing applications with high throughput requirements. Dataflow architecture has been considered a promising approach to optimize throughput since the intermediate feature map transfers can be significantly saved. However, previous MobileNetv2 accelerators only achieved a partial-dataflow architecture, and just one-third of the feature map transfers can be saved. To solve this issue, we propose a scheme to achieve a full-dataflow MobileNetv2 accelerator on FPGA. The scheme contains four techniques. First, we improve the full-integer quantization for easier deployment on hardware. Second, we propose tunable activation weight imbalance transfer for less quantization accuracy loss. Third, we present several highly optimized accelerator components whose parallelism can be flexibly adjusted, and implement residual connection with deeper FIFO so that the requirements of the full-dataflow architecture can be fully met. Finally, we present a computing resource allocation strategy to balance the latency of each layer, and a memory resource allocation strategy to effectively use the on-chip memory. Compared to the state-ofthe-art, experimental results show that the accelerator achieves 1910 FPS with 1.8× speedup when implemented on the Xilinx ZCU102 FPGA. In addition, it reaches 72.98% Top-1 accuracy with 8-bit integer quantization that outperforms all the other MobileNetv2 accelerators.

Study of FIT Dedicated Computer with Dataflow Architecture for High Performance 2-D Magneto-Static Field Simulation

An approach to dedicated computers is discussed in this study as a possibility for portable, low-cost, and low-power consumption high-performance computing technologies. Particularly, dataflow architecture dedicated computer of the finite integration technique (FIT) for 2D magnetostatic field simulation is considered for use in industrial applications. The dataflow architecture circuit of the BiCG-Stab matrix solver of the FIT matrix calculation is designed by the very high-speed integrated circuit hardware description language (VHDL). The operation of the dedicated computer's designed circuit is considered by VHDL logic circuit simulation.

Coordinated Cloud-Edge Anomaly Identification for Active Distribution Networks

The extensive integration of distributed energy resources transforms the conventional power distribution network into a complex and tightly-coupled cyber-physical system denoted as active distribution network (ADN). In this paper, a data-driven anomaly identification method based on the cloud-edge computing architecture is proposed to reduce the complicated risk of ADN operations. Especially, the proposed method would serve as a widely applicable and efficient line of defense against either cyber or physical anomalies. On one hand, a hierarchical data flow scheme is designed to balance the timeliness and economical requirements of anomaly identification by taking advantage of coordinated cloud-edge computing. On the other hand, an integrated algorithm is proposed on the basis of the proposed data flow architecture in order to identify anomalies with satisfactory robustness and accuracy, even under complex cyber-physical conditions. Results of numerical experiments have validated the computational superiority of the proposed method over the state-of-the-art by simulating a variety of physical faults and cyber attacks.

The Design of Efficient Data Flow and Low-Complexity Architecture for a Highly Configurable CNN Accelerator

The Design of Efficient Data Flow and Low-Complexity Architecture for a Highly Configurable CNN Accelerator