Domain-specific Accelerators Research Articles

Optimization of block-scaled integer GeMMs for efficient DNN deployment on scalable in-order vector processors

A continuing rise in DNN usage in distributed and embedded use cases has demanded more efficient hardware execution in the field. Low-precision GeMMs with optimized data formats have played a key role in more memory and computationally-efficient networks. Recently trending formats are block-scaled representations stemming from tight HW-SW co-optimization, that compress network size by sharing exponents per data block. Prior work mostly focuses on deploying such block-scaled GeMM operations on domain-specific accelerators for optimum efficiency at the cost of flexibility and ease of deployment. In this work, we exploit and optimize the deployment of block-scaled GeMMs on fully-programmable in-order vector processors using ARM SVE. We define a systematic methodology for performing design space exploration to optimally match the workload specifications with processor vector-lengths, different microkernels, block sizes and shapes. We introduce efficient intrinsics-based microkernels with effective loop unrollings, and data-transfer efficient fused requantization strategies to maximize kernel performance, while also ensuring several deployment configurations. We enable generalized block-scaled kernel deployments through tunable block sizes and shapes, which helps in accommodating different accuracy-speed trade-off requirements. Utilizing 2D activation blocks instead of conventional 1D blocks, the static and dynamic BS-INT8 configurations yielded on average 3.8x and 2.9x faster speedups over FP32 models respectively, at no accuracy loss for CNN classification tasks on CIFAR10/100 datasets.

GAHLS: an optimized graph analytics based high level synthesis framework

The urgent need for low latency, high-compute and low power on-board intelligence in autonomous systems, cyber-physical systems, robotics, edge computing, evolvable computing, and complex data science calls for determining the optimal amount and type of specialized hardware together with reconfigurability capabilities. With these goals in mind, we propose a novel comprehensive graph analytics based high level synthesis (GAHLS) framework that efficiently analyzes complex high level programs through a combined compiler-based approach and graph theoretic optimization and synthesizes them into message passing domain-specific accelerators. This GAHLS framework first constructs a compiler-assisted dependency graph (CaDG) from low level virtual machine (LLVM) intermediate representation (IR) of high level programs and converts it into a hardware friendly description representation. Next, the GAHLS framework performs a memory design space exploration while account for the identified computational properties from the CaDG and optimizing the system performance for higher bandwidth. The GAHLS framework also performs a robust optimization to identify the CaDG subgraphs with similar computational structures and aggregate them into intelligent processing clusters in order to optimize the usage of underlying hardware resources. Finally, the GAHLS framework synthesizes this compressed specialized CaDG into processing elements while optimizing the system performance and area metrics. Evaluations of the GAHLS framework on several real-life applications (e.g., deep learning, brain machine interfaces) demonstrate that it provides 14.27× performance improvements compared to state-of-the-art approaches such as LegUp 6.2.

FlowPix: Accelerating Image Processing Pipelines on an FPGA Overlay using a Domain Specific Compiler

The exponential performance growth guaranteed by Moore’s law has started to taper in recent years. At the same time, emerging applications like image processing demand heavy computational performance. These factors inevitably lead to the emergence of domain-specific accelerators (DSA) to fill the performance void left by conventional architectures. FPGAs are rapidly evolving towards becoming an alternative to custom ASICs for designing DSAs because of their low power consumption and a higher degree of parallelism. DSA design on FPGAs requires careful calibration of the FPGA compute and memory resources towards achieving optimal throughput. Hardware Descriptive Languages (HDL) like Verilog have been traditionally used to design FPGA hardware. HDLs are not geared towards any domain, and the user has to put in much effort to describe the hardware at the register transfer level. Domain Specific Languages (DSLs) and compilers have been recently used to weave together handwritten HDLs templates targeting a specific domain. Recent efforts have designed DSAs with image-processing DSLs targeting FPGAs. Image computations in the DSL are lowered to pre-existing templates or lower-level languages like HLS-C. This approach requires expensive FPGA re-flashing for every new workload. In contrast to this fixed-function hardware approach, overlays are gaining traction. Overlays are DSAs resembling a processor, which is synthesized and flashed on the FPGA once but is flexible enough to process a broad class of computations through soft reconfiguration. Less work has been reported in the context of image processing overlays. Image processing algorithms vary in size and shape, ranging from simple blurring operations to complex pyramid systems. The primary challenge in designing an image-processing overlay is maintaining flexibility in mapping different algorithms. This paper proposes a DSL-based overlay accelerator called FlowPix for image processing applications. The DSL programs are expressed as pipelines, with each stage representing a computational step in the overall algorithm. We implement 15 image-processing benchmarks using FlowPix on a Virtex-7-690t FPGA. The benchmarks range from simple blur operations to complex pipelines like Lucas-Kande optical flow. We compare FlowPix against existing DSL-to-FPGA frameworks like Hetero-Halide and Vitis Vision library that generate fixed-function hardware. On most benchmarks, we see up to 25% degradation in latency with approximately a 1.7x to 2x increase in the FPGA LUT consumption. Our ability to execute any benchmark without incurring the high costs of hardware synthesis, place-and-route, and FPGA re-flashing justifies the slight performance loss and increased resource consumption that we experience. FlowPix achieves an average frame rate of 170 FPS on HD frames of 1920 × 1080 pixels in the implemented benchmarks.

Symphony: Orchestrating Sparse and Dense Tensors with Hierarchical Heterogeneous Processing

Sparse tensor algorithms are becoming widespread, particularly in the domains of deep learning, graph and data analytics, and scientific computing. Current high-performance broad-domain architectures, such as GPUs, often suffer memory system inefficiencies by moving too much data or moving it too far through the memory hierarchy. To increase performance and efficiency, proposed domain-specific accelerators tailor their architectures to the data needs of a narrow application domain, but as a result cannot be applied to a wide range of algorithms or applications that contain a mix of sparse and dense algorithms. This article proposes Symphony, a hybrid programmable/specialized architecture that focuses on the orchestration of data throughout the memory hierarchy to simultaneously reduce the movement of unnecessary data and data movement distances. Key elements of the Symphony architecture include (1) specialized reconfigurable units aimed not only at roofline floating-point computations but also at supporting data orchestration features, such as address generation, data filtering, and sparse metadata processing; and (2) distribution of computation resources (both programmable and specialized) throughout the on-chip memory hierarchy. We demonstrate that Symphony can match non-programmable ASIC performance on sparse tensor algebra and provide 31× improved runtime and 44× improved energy over a comparably provisioned GPU for these applications.

QuickFPS: Architecture and Algorithm Co-Design for Farthest Point Sampling in Large-Scale Point Clouds

Point clouds have been employed extensively in machine perception applications. Farthest point sampling (FPS) is a critical kernel for point cloud processing. With the rapid growth of point cloud scale, FPS introduces a large number of memory accesses, which become the bottleneck of the large-scale point cloud processing. In this paper, we present QuickFPS, an architecture and algorithm co-design of farthest point sampling in large-scale point clouds. First, We systemically analyze the characteristics of FPS and put forward a bucket-based farthest point sampling algorithm. The algorithm introduces a two-level tree data structure to organize the large-scale point cloud into multiple buckets. By using two mechanisms named merged computation and implicit computation for the buckets, the external memory accesses and compute cost are significantly reduced. Then, we design an efficient domain-specific accelerator for farthest point sampling in large-scale point clouds. The accelerator takes advantage of different forms of parallelism and further improves the accelerator’s efficiency. Finally, we evaluate QuickFPS with several widely used point cloud datasets, which include small-scale and large-scale point clouds (up to 120,000 points). Overall, QuickFPS achieves performance speedups of 43.4x and 12.2x compared to GTX 1080Ti GPU and state-of-the-art point cloud accelerator PointAcc, respectively.

Cheshire: A Lightweight, Linux-Capable RISC-V Host Platform for Domain-Specific Accelerator Plug-In

Cheshire: A Lightweight, Linux-Capable RISC-V Host Platform for Domain-Specific Accelerator Plug-In

AutoMap: Automatic Mapping of Neural Networks to Deep Learning Accelerators for Edge Devices

Emerging deep neural networks (DNNs) have been emerging in applications (object detection, automatic speech recognition, etc.) deployed on edge devices. To improve the energy efficiency of edge devices, domain specific deep learning accelerators (DLAs) are designed with limited on-chip resources. The manifold DLA designs and evolving DNN topologies bring challenges for applications mapping and scheduling on hardware resources. In this paper, we propose an automatic DNN mapping framework named AutoMap given the hardware backend information. Firstly, a computational graph representation called Extended Directed Weighted Graph (EDWG) is proposed, which realizes unified expression for both spatial and temporal network inter-layer connections. Secondly, an associated partitioner is implemented for splitting an EDWG into subEDWGs, which incorporates the on-chip memory constraint and facilitates weight data reuse on chip. Lastly, a dynamic memory allocation strategy is utilized to alleviate the feature storing burden introduced by the multivarious network sizes and connections. Compared to the baseline mapping methods, experimental results show that our proposed automatic mapping framework can help to speedup the execution of several DNNs on state-of-the-art DLAs, ranging from 1.27x to 3.45x. The utilization of the PE array can increase from 20% to 64%.

High-Performance Acceleration of 2-D and 3-D CNNs on FPGAs Using Static Block Floating Point.

Over the past few years, 2-D convolutional neural networks (CNNs) have demonstrated their great success in a wide range of 2-D computer vision applications, such as image classification and object detection. At the same time, 3-D CNNs, as a variant of 2-D CNNs, have shown their excellent ability to analyze 3-D data, such as video and geometric data. However, the heavy algorithmic complexity of 2-D and 3-D CNNs imposes a substantial overhead over the speed of these networks, which limits their deployment in real-life applications. Although various domain-specific accelerators have been proposed to address this challenge, most of them only focus on accelerating 2-D CNNs, without considering their computational efficiency on 3-D CNNs. In this article, we propose a unified hardware architecture to accelerate both 2-D and 3-D CNNs with high hardware efficiency. Our experiments demonstrate that the proposed accelerator can achieve up to 92.4% and 85.2% multiply-accumulate efficiency on 2-D and 3-D CNNs, respectively. To improve the hardware performance, we propose a hardware-friendly quantization approach called static block floating point (BFP), which eliminates the frequent representation conversions required in traditional dynamic BFP arithmetic. Comparing with the integer linear quantization using zero-point, the static BFP quantization can decrease the logic resource consumption of the convolutional kernel design by nearly 50% on a field-programmable gate array (FPGA). Without time-consuming retraining, the proposed static BFP quantization is able to quantize the precision to 8-bit mantissa with negligible accuracy loss. As different CNNs on our reconfigurable system require different hardware and software parameters to achieve optimal hardware performance and accuracy, we also propose an automatic tool for parameter optimization. Based on our hardware design and optimization, we demonstrate that the proposed accelerator can achieve 3.8-5.6 times higher energy efficiency than graphics processing unit (GPU) implementation. Comparing with the state-of-the-art FPGA-based accelerators, our design achieves higher generality and up to 1.4-2.2 times higher resource efficiency on both 2-D and 3-D CNNs.

Toward Optimal Softcore Carry-aware Approximate Multipliers on Xilinx FPGAs

Domain-specific accelerators for signal processing, image processing, and machine learning are increasingly being implemented on SRAM-based field-programmable gate arrays (FPGAs). Owing to the inherent error tolerance of such applications, approximate arithmetic operations, in particular, the design of approximate multipliers, have become an important research problem. Truncation of lower bits is a widely used approximation approach; however, analyzing and limiting the effects of carry-propagation due to this approximation has not been explored in detail yet. In this article, an optimized carry-aware approximate radix-4 Booth multiplier design is presented that leverages the built-in slice look-up tables (LUTs) and carry-chain resources in a novel configuration. The proposed multiplier simplifies the computation of the upper and lower bits and provides significant benefits in terms of FPGA resource usage (LUTs saving 38.5%–42.9%), Power Delay Product (PDP saving 49.4%–53%), performance metric (LUTs × critical path delay (CPD) × PDP saving 68.9%–73.1%) and errors (70% improvement in mean relative error distance) compared to the latest state-of-the-art designs. Therefore, the proposed designs are an attractive choice to implement multiplication on FPGA-based accelerators.

Trireme: Exploration of Hierarchical Multi-level Parallelism for Hardware Acceleration

The design of heterogeneous systems that include domain specific accelerators is a challenging and time-consuming process. While taking into account area constraints, designers must decide which parts of an application to accelerate in hardware and which to leave in software. Moreover, applications in domains such as Extended Reality (XR) offer opportunities for various forms of parallel execution, including loop level, task level, and pipeline parallelism. To assist the design process and expose every possible level of parallelism, we present Trireme , a fully automated tool-chain that explores multiple levels of parallelism and produces domain-specific accelerator designs and configurations that maximize performance, given an area budget. FPGA SoCs were used as target platforms, and Catapult HLS [ 7 ] was used to synthesize RTL using a commercial 12 nm FinFET technology. Experiments on demanding benchmarks from the XR domain revealed a speedup of up to 20×, as well as a speedup of up to 37× for smaller applications, compared to software-only implementations.

A Domain-Specific Accelerator for Ultralow Latency Market Data Distribution System

Ultralow latency parsing of financial data is gaining significance in the high-frequency trading of the security exchange market. Hardware-aided systems exhibit superior improvement of latency over traditional software solutions, but flexibility may suffer when processing the financial protocol. This article presents a domain-specific accelerator for the market data distribution system, which integrates a financial information exchange adapted for streaming (FAST) decoder, a 10-Gbps network interface, and a high-speed PCIe host interface into a single field programmable gate array (FPGA) acceleration card. The proposed FAST decoder adopts the finite state machines-coordinated sequence mapping table to achieve run-time reconfigurability over fine-grained FPGA programming, and 16 fields can be decoded simultaneously in a pipelined manner, resulting in a decoding latency of only 33 ns. Evaluated on the Xilinx Alveo U200 acceleration card, this work improves the latency of decoding a FAST message by 26%–72% than state-of-the-art FPGA designs, and outperforms the software baseline by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&gt;27\times$</tex-math></inline-formula> in terms of latency covering both decoding and communication.

Unified Buffer: Compiling Image Processing and Machine Learning Applications to Push-Memory Accelerators

Image processing and machine learning applications benefit tremendously from hardware acceleration. Existing compilers target either FPGAs, which sacrifice power and performance for programmability, or ASICs, which become obsolete as applications change. Programmable domain-specific accelerators, such as coarse-grained reconfigurable arrays (CGRAs), have emerged as a promising middle-ground, but they have traditionally been difficult compiler targets since they use a different memory abstraction. In contrast to CPUs and GPUs, the memory hierarchies of domain-specific accelerators use push memories : memories that send input data streams to computation kernels or to higher or lower levels in the memory hierarchy and store the resulting output data streams. To address the compilation challenge caused by push memories, we propose that the representation of these memories in the compiler be altered to directly represent them by combining storage with address generation and control logic in a single structure—a unified buffer. The unified buffer abstraction enables the compiler to separate generic push memory optimizations from the mapping to specific memory implementations in the backend. This separation allows our compiler to map high-level Halide applications to different CGRA memory designs, including some with a ready-valid interface. The separation also opens the opportunity for optimizing push memory elements on reconfigurable arrays. Our optimized memory implementation, the Physical Unified Buffer, uses a wide-fetch, single-port SRAM macro with built-in address generation logic to implement a buffer with two read and two write ports. It is 18% smaller and consumes 31% less energy than a physical buffer implementation using a dual-port memory that only supports two ports. Finally, our system evaluation shows that enabling a compiler to support CGRAs leads to performance and energy benefits. Over a wide range of image processing and machine learning applications, our CGRA achieves 4.7× better runtime and 3.5× better energy-efficiency compared to an FPGA.

Towards Machine Learning-Based FPGA Backend Flow: Challenges and Opportunities

Field-Programmable Gate Array (FPGA) is at the core of System on Chip (SoC) design across various Industry 5.0 digital systems—healthcare devices, farming equipment, autonomous vehicles and aerospace gear to name a few. Given that pre-silicon verification using Computer Aided Design (CAD) accounts for about 70% of the time and money spent on the design of modern digital systems, this paper summarizes the machine learning (ML)-oriented efforts in different FPGA CAD design steps. With the recent breakthrough of machine learning, FPGA CAD tasks—high-level synthesis (HLS), logic synthesis, placement and routing—are seeing a renewed interest in their respective decision-making steps. We focus on machine learning-based CAD tasks to suggest some pertinent research areas requiring more focus in CAD design. The development of open-source benchmarks optimized for an end-to-end machine learning experience, intra-FPGA optimization, domain-specific accelerators, lack of explainability and federated learning are the issues reviewed to identify important research spots requiring significant focus. The potential of the new cloud-based architectures to understand the application of the right ML algorithms in FPGA CAD decision-making steps is discussed, together with visualizing the scenario of incorporating more intelligence in the cloud platform, with the help of relatively newer technologies such as CAD as Adaptive OpenPlatform Service (CAOS). Altogether, this research explores several research opportunities linked with modern FPGA CAD flow design, which will serve as a single point of reference for modern FPGA CAD flow design.

VOTA: A Heterogeneous Multicore Visual Object Tracking Accelerator Using Correlation Filters

A correlation filter (CF) is a lighter and faster solution for visual object tracking (VOT) compared with a deep neural network (DNN). However, the CF-based trackers introduce diverse computational kernels and require FP16 computations. To address these challenges, we present a VOT accelerator (VOTA), a domain-specific accelerator for CF-based VOT that meets real-time performance at high efficiency, while providing flexibility and programmability. The VOTA encompasses a Winograd convolution core (WINO), a fast Fourier transformation (FFT) core (FFT), and a vector core (VEC) for diverse kernels, integrated in a high-bandwidth star-ring topology. The VOTA’s frame-based instruction set and execution enable a 537 GFLOPS performance, adapt to variances of CF trackers, and reduce the code size. An instruction-chaining mechanism permits inter-core pipelining and improves the hardware utilization up to 84.2%. A 10.2-mm 2 28-nm FP16 system on chip (SoC) prototype incorporating the VOTA, an RISC-V host CPU, and other supportive peripherals is taped out and measured to achieve 2.45 TFLOPS/W at 0.72 V. Running the oriented particle CF (OPCF), a CF tracker enhanced by adaptive boosting and particle filtering, the SoC chip achieves 1157 frames/s ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$640\times480$ </tex-math></inline-formula> frame size) at 0.9 V and 175 MHz, consuming 296 mW.

Optimization of a line detection algorithm for autonomous vehicles on a RISC-V with accelerator

In recent years, autonomous vehicles have attracted theattention of many research groups, both in academiaand business, including researchers from leading com-panies such as Google, Uber and Tesla. This type ofvehicles are equipped with systems that are subjectto very strict requirements, essentially aimed at per-forming safe operations –both for potential passengersand pedestrians– as well as carrying out the process-ing needed for decision making in real time. In manyinstances, general-purpose processors alone cannotensure that these safety, reliability and real-time re-quirements are met, so it is common to implementheterogeneous systems by including accelerators. Thispaper explores the acceleration of a line detection ap-plication in the autonomous car environment using aheterogeneous system consisting of a general-purposeRISC-V core and a domain-specific accelerator. In par-ticular, the application is analyzed to identify the mostcomputationally intensive parts of the code and it isadapted accordingly for more efficient processing. Fur-thermore, the code is executed on the aforementionedhardware platform to verify that the execution effec-tively meets the existing requirements in autonomousvehicles, experiencing a 3.7x speedup with respect torunning without accelerator.

Methodology for Structured Data-Path Implementation in VLSI Physical Design: A Case Study

State-of-the-art modern microprocessor and domain-specific accelerator designs are dominated by data-paths composed of regular structures, also known as bit-slices. Random logic placement and routing techniques may not result in an optimal layout for these data-path-dominated designs. As a result, implementation tools such as Cadence’s Innovus include a Structured Data-Path (SDP) feature that allows data-path placement to be completely customized by constraining the placement engine. A relative placement file is used to provide these constraints to the tool. However, the tool neither extracts nor automatically places the regular data-path structures. In other words, the relative placement file is not automatically generated. In this paper, we propose a semi-automated method for extracting bit-slices from the Innovus SDP flow. It has been demonstrated that the proposed method results in 17% less density or use for a pixel buffer design. At the same time, the other performance metrics are unchanged when compared to the traditional place and route flow.

HPVM: Hardware-Agnostic Programming for Heterogeneous Parallel Systems

We present Heterogeneous Parallel Virtual Machine (HPVM), a compiler framework for hardware-agnostic programming on heterogeneous compute platforms. HPVM introduces a hardware-agnostic parallel intermediate representation with constructs for the hierarchical task, data, and pipeline parallelism, including dataflow parallelism, and supports multiple front-end languages. In addition, HPVM provides optimization passes that navigate performance, energy, and accuracy tradeoffs, and includes retargetable back ends for a wide range of diverse hardware targets, including central processing units, graphics processing units, domain-specific accelerators, and field-programmable gate arrays. Across diverse hardware platforms, HPVM optimizations provide significant performance and energy improvements, while preserving object-code portability. With these capabilities, HPVM facilitates developers, domain experts, and hardware vendors in programming modern heterogeneous systems.

Reconfigurable and hardware efficient adaptive quantization model-based accelerator for binarized neural network

Binarized neural networks (BNNs) architecture play a vital role in the development of deep learning accelerator for memory-constrained IoT devices. However, the cost-efficiency of the domain-specific accelerators still requires a simplified structure to enhance the computing performance of BNNs. We propose and present a reconfigurable BNN accelerator to improve the computing speed through channel amplitude and an adaptive spatial amplitude model. To reduce the redundant operations in the binarization of matrix multiplication, we introduce the channel amplitude model for BNN convolution functions. The adaptive spatial amplitude is implemented with the help of a matrix multiplication layer with 3 × 3 convolutions to get spatial information and increases the computation speed. For hardware implementation, the channel amplitude conversion is deployed through XNOR-popcount convolutions of each layer. The performance results of our proposed accelerator provide 2–10 × line buffer efficiency, 1.6–3.7 × processing frequency enhancement, and 2.35 × processing element reduction compared with existing baselines using VGG-16 based FPGA implementation.

Analysis of an Application Specific Instruction-set Processor's expansion potential, considering performance and implementation effort

Modern embedded systems need to provide high compute power while being energy efficient. This can be achieved by domain specific processing cores and accelerators that have been optimized for a specific class of applications. The optimizations can be introduced in the structure of the memory system, the instruction set architecture, special instructions that are provided and in the interfaces that allow the coupling of external accelerators. In this work we investigated the opportunities to adopt a processor by introducing different instruction set extensions, the creation of manually designed instruction extensions for special applications and the extension of the core by a Very Large Instruction Word (VLIW) extension. All modifications have been analyzed using a benchmark suite with applications from different domains. In addition to performance measurements, we rated the effort, that specific extensions introduce to the hard- and the software-designer. All experiments have been done using the Cadence Tensilica Xtensa LX Application Specific Instruction-set Processor (ASIP) in the cycle-accurate simulation mode. Or work shows, that speedups of 4x can be reached by just adding around 10% of gates to the processor. For this kind of analysis, we show an evaluation methodology and introduce a metric.

Algorithm-Architecture Co-Design for Domain-Specific Accelerators in Communication and Artificial Intelligence

Algorithm-Architecture Co-Design for Domain-Specific Accelerators in Communication and Artificial Intelligence