Systolic Array Research Articles

Systolic array-based CNN accelerator soft error approximate fault tolerance design

To satisfy the massive computational requirement of Convolutional Neural Networks, various Domain-Specific Architecture based accelerators have been deployed in large-scale systems. While improving the performance significantly, the high integration of the accelerator makes it much more susceptible to soft-error, which will be propagated and amplified layer by layer during the execution of CNN, finally disturbing the decision of CNN and leading to catastrophic consequences. CNNs have been increasingly deployed in security-critical areas, requiring more attention to reliable execution. Although the classical fault-tolerant approaches are error-effective, the performance/energy overheads introduced are non-negligible, which is the opposite of CNN accelerator design philosophy. In this article, we leverage CNN's intrinsic tolerance for minor errors and the similarity of filters within a layer to explore the Approximate Fault Tolerance opportunities for CNN accelerator fault tolerance overhead reduction. By gathering the filters into several check groups by clustering to perform an inexact check while ensuring that serious errors are mitigated, our approximate fault tolerance design can reduce fault tolerance overhead significantly. Furthermore, we remap the filters to match the checking process and the dataflow of systolic array, which can satisfy the real-time checking demands of CNN. Experimental results exhibit that our approach can reduce 73.39%performance degradation of baseline DMR.

A certain examination on heterogeneous systolic array (HSA) design for deep learning accelerations with low power computations

Acceleration techniques play a crucial role in enhancing the performance of modern high-speed computations, especially in Deep Learning (DL) applications where the speed is of utmost importance. One essential component in this context is the Systolic Array (SA), which effectively handles computational tasks and data processing in a rhythmic manner. Google's Tensor Processing Unit (TPU) leverages the power of SA for neural networks. The core SA's functionality and performance lies in the Computation Element (CE), which facilitates parallel data flow. In our article, we introduce a novel approach called Proposed Systolic Array (PSA), which is implemented on the CE and further enhanced with a modified Hybrid Kogge Stone adder (MHA). This design incorporates principles to expedite computations by rounding and extracting data model in SA as PSA-MHA. The PSA, utilizing a data flow model with MHA, significantly accelerates data shifts and control passes in execution cycles. We validated our approach through simulations on the Cadence Virtuoso platform using 65 nm process technology, comparing it to the General Matrix Multiplication (GMMN) benchmark. The results showed remarkable improvements in the CE, with a 30.29 % reduction in delay, a 23.07 % reduction in area, and an 11.87 % reduction in power consumption. The PSA outperformed these improvements, achieving a 46.38 % reduction in delay, a 7.58 % reduction in area, and an impressive 48.23 % decrease in Area Delay Product (ADP). To further substantiate our findings, we applied the PSA-based approach to pre-trained hybrid Convolutional and Recurrent (CNN-RNN) neural models. The PSA-based hybrid model incorporates 189 million Multiply-Accumulate (MAC) units, resulting in a weighted mean architecture value of 784.80 for the RNN component. We also explored variations in bit width, which led to delay reductions ranging from 20.17 % to 30.29 %, area variations between 13.08 % and 32.16 %, and power consumption changes spanning from 11.88 % to 20.42 %.

Accelerating Neural Network Inference in Handwritten Digit Recognition — Comparative Study

In deep learning, one of the popular subclasses of multilayer perceptron is the Convolutional Neural Networks (CNNs), optimized for image and semantic segmentation tasks. With a large number of floating-point operations and relatively less data transfer in the training phase, CNNs are well suited to be handled by parallel architectures. The high accuracy of CNNs comes at the cost of consequential compute and memory demands. In this article, we present a comparative analysis of a pre-trained CNN framework for recognizing a handwritten digit on different processing platforms. The performance of the neural network as the function of the parallel processing platforms offers parametric insights and factors that influence the inference on state-of-the-art Graphics Processing Units (GPUs) and systolic arrays such as Eyeriss and Tensor Processing Unit (TPU). Through inference time analysis, we observed that the systolic arrays can outperform upto 58.7 times better than a Turing architecture powered GPU. We show that while the efficiency with which the available resources are utilized is higher in the existing GPUs (upto 32%) than the TPUs, the efficiency of application-specific systolic arrays can be on par with that of GPUs. We present the results obtained from three different customized systolic array-based platforms which can be adopted by the designers to decide the hardware optimization goal.

Unifying Static and Dynamic Intermediate Languages for Accelerator Generators

Compilers for accelerator design languages (ADLs) translate high-level languages into application-specific hardware. ADL compilers rely on a hardware control interface to compose hardware units. There are two choices: static control, which relies on cycle-level timing; or dynamic control, which uses explicit signalling to avoid depending on timing details. Static control is efficient but brittle; dynamic control incurs hardware costs to support compositional reasoning. Piezo is an ADL compiler that unifies static and dynamic control in a single intermediate language (IL). Its key insight is that the IL’s static fragment is a refinement of its dynamic fragment: static code admits a subset of the run-time behaviors of the dynamic equivalent. Piezo can optimize code by combining facts from static and dynamic submodules, and it opportunistically converts code from dynamic to static control styles. We implement Piezo as an extension to an existing dynamic ADL compiler, Calyx. We use Piezo to implement a frontend for an existing ADL, a systolic array generator, and a packet-scheduling hardware generator to demonstrate its optimizations and the static–dynamic interactions it enables.

High-throughput systolic array-based accelerator for hybrid transformer-CNN networks

In this era of Transformers enjoying remarkable success, Convolutional Neural Networks (CNNs) remain highly relevant and useful. Indeed, hybrid Transformer-CNN network architectures, which combine the benefits of both approaches, have achieved impressive results. Vision Transformer (ViT) is a significant neural network architecture that features a convolutional layer as its first layer, primarily built on the transformer framework. However, owing to the distinct computation patterns inherent in attention and convolution, existing hardware accelerators for these two models are typically designed separately and lack a unified approach toward accelerating both models efficiently. In this paper, we present a dedicated accelerator on a field-programmable gate array (FPGA) platform. The accelerator, which integrates a configurable three-dimensional systolic array, is specifically designed to accelerate the inferential capabilities of hybrid Transformer-CNN networks. The Convolution and Transformer computations can be mapped to a systolic array by unifying these operations for matrix multiplication. Softmax and LayerNorm which are frequently used in hybrid Transformer-CNN networks were also implemented on FPGA boards. The accelerator achieved high performance with a peak throughput of 722 GOP/s at an average energy efficiency of 53 GOPS/W. Its respective computation latencies were 51.3 ms, 18.1 ms, and 6.8 ms for ViT-Base, ViT-Small, and ViT-Tiny. The accelerator provided a 12× improvement in energy efficiency compared to the CPU, a 2.3× improvement compared to the GPU, and a 1.5× to 2× improvement compared to existing accelerators regarding speed and energy efficiency.

Enhancing efficiency in spaceborne phased array systems: MVDR algorithm and FPGA integration

Digital Beamforming (DBF) has garnered significant attention within the space community, driven by the heightened flexibility offered by satellites equipped with active antennas for diverse applications like spaceborne synthetic aperture radar (SAR) systems and communication services. This paper presents a comprehensive exploration encompassing analysis, design, and implementation of a Minimum Variance Distortionless Response (MVDR) algorithm tailored for application in a Digital Beamforming Receiver system. The described system leverages the capabilities of the high-speed Analog to Digital Converter (ADC) device EV12AQ600, space-qualified, and the high-performance FPGA XCKU085. These components are seamlessly integrated into the EV12AQ60X-ADX-EVM evaluation board, thoughtfully provided by Teledyne e2v Semiconductors. The design methodology outlined herein is driven by the overarching objective to minimize FPGA resource utilization and power consumption for spaceborne phased array systems. This objective is chiefly met through the implementation of a systolic array structure adept at processing both real and complex input samples. This innovative approach results in a substantial reduction in allocated resources compared to conventional architectures. Furthermore, the design incorporates the use of the CORDIC algorithm to compute the inverse of the correlation matrix, obviating the need for square root and division operations. This strategic utilization of algorithms enhances the system's efficiency in terms of resource utilization, computational load, and processing time. To validate the anticipated behavior of the adaptive beamforming system, various radio frequency (RF) input signals are applied to the system, which is physically instantiated on the EV12AQ60X-ADX-EVM evaluation board. The experimental results affirm the efficacy of the proposed design, underscoring its suitability for spaceborne applications.

ReDas: A Lightweight Architecture for Supporting Fine-Grained Reshaping and Multiple Dataflows on Systolic Array

ReDas: A Lightweight Architecture for Supporting Fine-Grained Reshaping and Multiple Dataflows on Systolic Array

Enhancing Computation-Efficiency of Deep Neural Network Processing on Edge Devices through Serial/Parallel Systolic Computing

In recent years, deep neural networks (DNNs) have addressed new applications with intelligent autonomy, often achieving higher accuracy than human experts. This capability comes at the expense of the ever-increasing complexity of emerging DNNs, causing enormous challenges while deploying on resource-limited edge devices. Improving the efficiency of DNN hardware accelerators by compression has been explored previously. Existing state-of-the-art studies applied approximate computing to enhance energy efficiency even at the expense of a little accuracy loss. In contrast, bit-serial processing has been used for improving the computational efficiency of neural processing without accuracy loss, exploiting a simple design, dynamic precision adjustment, and computation pruning. This research presents Serial/Parallel Systolic Array (SPSA) and Octet Serial/Parallel Systolic Array (OSPSA) processing elements for edge DNN acceleration, which exploit bit-serial processing on systolic array architecture for improving computational efficiency. For evaluation, all designs were described at the RTL level and synthesized in 28 nm technology. Post-synthesis cycle-accurate simulations of image classification over DNNs illustrated that, on average, a sample 16 × 16 systolic array indicated remarkable improvements of 17.6% and 50.6% in energy efficiency compared to the baseline, with no loss of accuracy.

Factored Systolic Arrays Based on Radix-8 Multiplication for Machine Learning Acceleration

Factored Systolic Arrays Based on Radix-8 Multiplication for Machine Learning Acceleration

Design and implementation of a nano-scale high-speed multiplier for signal processing applications

Digital signal processing (DSP) is an engineering field involved with increasing the precision and dependability of digital communications and mathematical processes, including equalization, modulation, demodulation, compression, and decompression, which can be used to produce a signal of the highest caliber. To execute vital tasks in DSP, an essential electronic circuit such as a multiplier plays an important role, continually performing tasks such as the multiplication of two binary numbers. Multiplier is a crucial component utilized to implement a wide range of DSP tasks, including convolution, Fourier transform, discrete wavelet transforms (DWT), filtering and dithering, multimedia information processing, and more. A multiplier device includes a clock and reset buttons for more flexible operational control. Each digital signal processor constitutes a multiplier unit. A multiplier unit functions entirely autonomously from the central processing unit (CPU); consequently, the CPU is burdened with a significantly reduced amount of work. Since DSP algorithms must constantly carry out multiplication tasks, the employment of a high-speed multiplier to execute fast-speed filtering processes is vital. The previous multipliers had lots of weaknesses, such as high energy, low speed, and high area, because they implemented this necessary circuit based on traditional technology such as complementary metal-oxide semiconductor (CMOS) and very large-scale integration (VLSI). To solve all previous drawbacks in this necessary circuit, we can use nanotechnology, which directly affects the performance of the multiplier and can overcome all previous issues. One of the alternative nanotechnologies that can be used for designing digital circuits is quantum dot cellular automata, which is high speed, low area, and low power. Therefore, this manuscript suggests a quantum technology-based multiplier for DSP applications. In addition, some vital circuits, such as half adder, full adder, and ripple carry adder (RCA), are suggested for designing a multiplier. Moreover, a systolic array, accumulator, and multiply and accumulate (MAC) unit are proposed based on the quantum technology-based multiplier. Nonetheless, each of the suggested frameworks has a coplanar configuration without rotated cells. The suggested structure is developed and verified utilizing the QCADesigner 2.0.3 tools. The findings showed that all circuits have no complicated configuration, including a higher number of quantum cells, latency, and an optimum area.

FPGA-Based Acceleration of Polar-Format Algorithm for Video Synthetic-Aperture Radar Imaging

This paper presents a polar-format algorithm (PFA)-based synthetic-aperture radar (SAR) processor that can be mounted on a small drone to support video SAR (ViSAR) imaging. For drone mounting, it requires miniaturization, low power consumption, and high-speed performance. Therefore, to meet these requirements, the processor design was based on a field-programmable gate array (FPGA), and the implementation results are presented. The proposed PFA-based SAR processor consists of both an interpolation unit and a fast Fourier transform (FFT) unit. The interpolation unit uses linear interpolation for high speed while occupying a small space. In addition, the memory transfer is minimized through optimized operations using SAR system parameters. The FFT unit uses a base-4 systolic array architecture, chosen from among various fast parallel structures, to maximize the processing speed. Each unit is designed as a reusable block (IP core) to support reconfigurability and is interconnected using the advanced extensible interface (AXI) bus. The proposed PFA-based SAR processor was designed using Verilog-HDL and implemented on a Xilinx UltraScale+ MPSoC FPGA platform. It generates an image 2048 × 2048 pixels in size within 0.766 s, which is 44.862 times faster than that achieved by the ARM Cortex-A53 microprocessor. The speed-to-area ratio normalized by the number of resources shows that it achieves a higher speed at lower power consumption than previous studies.

Symmetry-Enabled Resource-Efficient Systolic Array Design for Montgomery Multiplication in Resource-Constrained MIoT Endpoints

In today’s TEST interconnected world, the security of 5G Medical IoT networks is of paramount concern. The increasing number of connected devices and the transmission of vast amounts of data necessitate robust measures to protect information integrity and confidentiality. However, securing Medical IoT edge nodes poses unique challenges due to their limited resources, making the implementation of cryptographic protocols a complex task. Within these protocols, modular multiplication assumes a crucial role. Therefore, careful consideration must be given to its implementation. This study focuses on developing a resource-efficient hardware implementation of the Montgomery modular multiplication algorithm over GF(2l), which is a critical operation in cryptographic algorithms. The proposed solution introduces a bit-serial systolic array layout with a modular structure and local connectivity between processing elements. This design, inspired by the principles of symmetry, allows for efficient utilization of resources and optimization of area and delay management. This makes it well-suited for deployment in compact Medical IoT edge nodes with limited resources. The suggested bit-serial processor structure was evaluated through ASIC implementation, which demonstrated substantial improvements over competing designs. The results showcase an average area reduction of 24.5% and significant savings in the area–time product of 26.2%.

Enhancing Field Multiplication in IoT Nodes with Limited Resources: A Low-Complexity Systolic Array Solution

Enhancing Field Multiplication in IoT Nodes with Limited Resources: A Low-Complexity Systolic Array Solution

Highly Fault-Tolerant Systolic-Array-Based Matrix Multiplication

Matrix multiplication plays a crucial role in various engineering and scientific applications. Cannon’s algorithm, executed within two-dimensional systolic arrays, significantly enhances computational efficiency through parallel processing. However, as the matrix size increases, reliability issues become more prominent. Although the previous work has proposed a fault-tolerant mechanism, it is only suitable for scenarios with a limited number of faulty processing elements (PEs). This paper introduces a pair-matching mechanism, assigning a fault-free PE as a proxy for each faulty PE to execute its tasks. Our fault-tolerant mechanism comprises two stages: in the first stage, each fault-free PE completes its designated computations; in the second stage, computations intended for each faulty PE are executed by its assigned fault-free PE proxy. The experimental results demonstrate that compared to the previous work, our approach not only significantly improves the fault tolerance of systolic arrays (applicable to scenarios with a higher number of faulty PEs) but also reduces circuit areas. Therefore, the proposed approach proves effective in practical applications.

High-Speed CNN Accelerator SoC Design Based on a Flexible Diagonal Cyclic Array

The latest convolutional neural network (CNN) models for object detection include complex layered connections to process inference data. Each layer utilizes different types of kernel modes, so the hardware needs to support all kernel modes at an optimized speed. In this paper, we propose a high-speed and optimized CNN accelerator with flexible diagonal cyclic arrays (FDCA) that supports the acceleration of CNN networks with various kernel sizes and significantly reduces the time required for inference processing. The accelerator uses four FDCAs to simultaneously calculate 16 input channels and 8 output channels. Each FDCA features a 4 × 8 systolic array that contains a 3 × 3 processing element (PE) array and is designed to handle the most commonly used kernel sizes. To evaluate the proposed CNN accelerator, we mapped the widely used YOLOv5 CNN model and evaluated the performance of its implementation on the Zynq UltraScale+ MPSoC ZCU102 FPGA. The design consumes 249,357 logic cells, 2304 DSP blocks, and only 567 KB BRAM. In our evaluation, the YOLOv5n model achieves an accuracy of 43.1% (mAP@0.5). A prototype accelerator has been implemented using Samsung’s 14 nm CMOS technology. It achieves 1.075 TOPS, a peak performance with a 400 MHz clock frequency.

VerSA: Versatile Systolic Array Architecture for Sparse and Dense Matrix Multiplications

A key part of modern deep neural network (DNN) applications is matrix multiplication. As DNN applications are becoming more diverse, there is a need for both dense and sparse matrix multiplications to be accelerated by hardware. However, most hardware accelerators are designed to accelerate either dense or sparse matrix multiplication. In this paper, we propose VerSA, a versatile systolic array architecture for both dense and sparse matrix multiplications. VerSA employs intermediate paths and SRAM buffers between the rows of the systolic array (SA), thereby enabling an early termination in sparse matrix multiplication with a negligible performance overhead when running dense matrix multiplication. When running sparse matrix multiplication, 256 × 256 VerSA brings performance (i.e., an inverse of execution time) improvement and energy saving by 1.21×–1.60× and 7.5–30.2%, respectively, when compared to the conventional SA. When running dense matrix multiplication, VerSA results in only a 0.52% performance overhead compared to the conventional SA.

Voltage Scaled Low Power DNN Accelerator Design on Reconfigurable Platform

The exponential emergence of Field-Programmable Gate Arrays (FPGAs) has accelerated research on hardware implementation of Deep Neural Networks (DNNs). Among all DNN processors, domain-specific architectures such as Google’s Tensor Processor Unit (TPU) have outperformed conventional GPUs (Graphics Processing Units) and CPUs (Central Processing Units). However, implementing low-power TPUs in reconfigurable hardware remains a challenge in this field. Voltage scaling, a popular approach for energy savings, can be challenging in FPGAs, as it may lead to timing failures if not implemented appropriately. This work presents an ultra-low-power FPGA implementation of a TPU for edge applications. We divide the systolic array of a TPU into different FPGA partitions based on the minimum slack value of different design paths of Multiplier Accumulators (MACs). Each partition uses different near-threshold (NTC) biasing voltages to run its FPGA cores. The biasing voltage for each partition is roughly calculated by the proposed static schemes. However, further calibration of biasing voltage is performed by the proposed runtime scheme. To overcome the timing failure caused by NTC, the MACs with higher minimum slack are placed in lower-voltage partitions, while the MACs with lower minimum slack paths are placed in higher-voltage partitions. The proposed architecture is implemented in a commercial platform, namely Vivado with Xilinx Artix-7 FPGA and academic platform VTR with 22 nm, 45 nm and 130 nm FPGAs. Any timing error caused by NTC can be caught by the Razor flipflop used in each MAC. The proposed voltage-scaled, partitioned systolic array can save 3.1% to 11.6% of dynamic power in Vivado and VTR tools, respectively, depending on the FPGA technology, partition size, number of partitions and biasing voltages. The normalized performance and accuracy of benchmark models running on our low-power TPU are very competitive compared to existing literature.

Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models

Current AI training infrastructure is dominated by single instruction multiple data (SIMD) and systolic array architectures, such as Graphics Processing Units (GPUs) and Tensor Processing Units (TPUs), that excel at accelerating parallel workloads and dense vector matrix multiplications. Potentially more efficient neural network models utilizing sparsity and recurrence cannot leverage the full power of SIMD processor and are thus at a severe disadvantage compared to today's prominent parallel architectures like Transformers and CNNs, thereby hindering the path towards more sustainable AI. To overcome this limitation, we explore sparse and recurrent model training on a massively parallel multiple instruction multiple data (MIMD) architecture with distributed local memory. We implement a training routine based on backpropagation though time (BPTT) for the brain-inspired class of Spiking Neural Networks (SNNs) that feature binary sparse activations. We observe a massive advantage in using sparse activation tensors with a MIMD processor, the Intelligence Processing Unit (IPU) compared to GPUs. On training workloads, our results demonstrate 5-10x throughput gains compared to A100 GPUs and up to 38x gains for higher levels of activation sparsity, without a significant slowdown in training convergence or reduction in final model performance. Furthermore, our results show highly promising trends for both single and multi IPU configurations as we scale up to larger model sizes. Our work paves the way towards more efficient, non-standard models via AI training hardware beyond GPUs, and competitive large scale SNN models.

ReSA: Reconfigurable Systolic Array for Multiple Tiny DNN Tensors

Systolic array architecture has significantly accelerated deep neural networks (DNNs). A systolic array comprises multiple processing elements (PEs) that can perform multiply-accumulate (MAC). Traditionally, the systolic array can execute a certain amount of tensor data that matches the size of the systolic array simultaneously at each cycle. However, hyper-parameters of DNN models differ across each layer and result in various tensor sizes in each layer. Mapping these irregular tensors to the systolic array while fully utilizing the entire PEs in a systolic array is challenging. Furthermore, modern DNN systolic accelerators typically employ a single dataflow. However, such a dataflow isn’t optimal for every DNN model. This work proposes ReSA, a reconfigurable dataflow architecture that aims to minimize the execution time of a DNN model by mapping tiny tensors on the spatially partitioned systolic array. Unlike conventional systolic array architectures, the ReSA data path controller enables the execution of the input, weight, and output-stationary dataflow on PEs. ReSA also decomposes the coarse-grain systolic array into multiple small ones to reduce the fragmentation issue on the tensor mapping. Each small systolic sub-array unit relies on our data arbiter to dispatch tensors to each other through the simple interconnected network. Furthermore, ReSA reorders the memory access to overlap the memory load and execution stages to hide the memory latency when tackling tiny tensors. Finally, ReSA splits tensors of each layer into multiple small ones and searches for the best dataflow for each tensor on the host side. Then, ReSA encodes the predefined dataflow in our proposed instruction to notify the systolic array to switch the dataflow correctly. As a result, our optimization on the systolic array architecture achieves a geometric mean speedup of 1.87X over the weight-stationary systolic array architecture across 9 different DNN models.

Efficient ASIC Architecture for Low Latency Classic McEliece Decoding

Post-quantum cryptography addresses the increasing threat that quantum computing poses to modern communication systems. Among the available “quantum-resistant” systems, the Classic McEliece key encapsulation mechanism (KEM) is positioned as a conservative choice with strong security guarantees. Building upon the code-based Niederreiter cryptosystem, this KEM enables high performance encapsulation and decapsulation and is thus ideally suited for applications such as the acceleration of server workloads. However, until now, no ASIC architecture is available for low latency computation of Classic McEliece operations. Therefore, the present work targets the design, implementation and optimization of a tailored ASIC architecture for low latency Classic McEliece decoding. An efficient ASIC design is proposed, which was implemented and manufactured in a 22 nm FDSOI CMOS technology node. We also introduce a novel inversionless architecture for the computation of error-locator polynomials as well as a systolic array for combined syndrome computation and polynomial evaluation. With these approaches, the associated optimized architecture improves the latency of computing error-locator polynomials by 47% and the overall decoding latency by 27% compared to a state-of-the-art reference, while requiring only 25% of the area.