Convolutional Neural Network Accelerator Research Articles

NAND-SPIN-based processing-in-MRAM architecture for convolutional neural network acceleration

The performance and efficiency of running large-scale datasets on traditional computing systems exhibit critical bottlenecks due to the existing "power wall" and "memory wall" problems. To resolve those problems, processing-in-memory (PIM) architectures are developed to bring computation logic in or near memory to alleviate the bandwidth limitations during data transmission. NAND-like spintronics memory (NAND-SPIN) is one kind of promising magnetoresistive random-access memory (MRAM) with low write energy and high integration density, and it can be employed to perform efficient in-memory computation operations. In this work, we propose a NAND-SPIN-based PIM architecture for efficient convolutional neural network (CNN) acceleration. A straightforward data mapping scheme is exploited to improve the parallelism while reducing data movements. Benefiting from the excellent characteristics of NAND-SPIN and in-memory processing architecture, experimental results show that the proposed approach can achieve $\sim$2.6$\times$ speedup and $\sim$1.4$\times$ improvement in energy efficiency over state-of-the-art PIM solutions.

Area Efficient Compression for Floating-Point Feature Maps in Convolutional Neural Network Accelerators

Since Convolutional neural networks (CNNs) need massive computing resources, lots of computing architectures are proposed to improve the throughput and energy efficiency of the computing. However, those computing architectures need high data movement between the chip and off-chip memories, which causes high energy consumption on the off-chip memory; thus, the feature map (fmap) compression has been discussed for reducing the data movement. Therefore, the design of fmap compression becomes one of the main researches on CNN accelerator for energy efficiency on the off-chip memory. In this brief, we proposed floating-point (FP) fmap compression for a hardware accelerator which includes hardware design and a compression algorithm. This can apply quantization methods such as ternary neural quantization (TTQ), which only quantized weights with little or no degradation in accuracy and reduces the computation cost. In addition to the zero compression, we also compress nonzero values in the fmap based on the FP format. The compression algorithm achieves low area overhead and a similar compression ratio compared with the state-of-the-art on ILSVRC 2012 dataset.

GShuttle: Optimizing Memory Access Efficiency for Graph Convolutional Neural Network Accelerators

GShuttle: Optimizing Memory Access Efficiency for Graph Convolutional Neural Network Accelerators

Efficient Binary Weight Convolutional Network Accelerator for Speech Recognition.

Speech recognition has progressed tremendously in the area of artificial intelligence (AI). However, the performance of the real-time offline Chinese speech recognition neural network accelerator for edge AI needs to be improved. This paper proposes a configurable convolutional neural network accelerator based on a lightweight speech recognition model, which can dramatically reduce hardware resource consumption while guaranteeing an acceptable error rate. For convolutional layers, the weights are binarized to reduce the number of model parameters and improve computational and storage efficiency. A multichannel shared computation (MCSC) architecture is proposed to maximize the reuse of weight and feature map data. The binary weight-sharing processing engine (PE) is designed to avoid limiting the number of multipliers. A custom instruction set is established according to the variable length of voice input to configure parameters for adapting to different network structures. Finally, the ping-pong storage method is used when the feature map is an input. We implemented this accelerator on Xilinx ZYNQ XC7Z035 under the working frequency of 150 MHz. The processing time for 2.24 s and 8 s of speech was 69.8 ms and 189.51 ms, respectively, and the convolution performance reached 35.66 GOPS/W. Compared with other computing platforms, accelerators perform better in terms of energy efficiency, power consumption and hardware resource consumption.

Accelerating Deep Convolutional Neural Networks Using Number Theoretic Transform

Modern deep convolutional neural networks (CNNs) suffer from high computational complexity due to excessive convolution operations. Recently, fast convolution algorithms such as fast Fourier transform (FFT) and Winograd transform have gained attention to address this problem. They reduce the number of multiplications required in the convolution operation by replacing it with element-wise multiplication in the transform domain. However, fast convolution-based CNN accelerators have three major concerns: expensive domain transform, large memory overhead, and limited flexibility in kernel size. In this paper, we present a novel CNN accelerator based on number theoretic transform (NTT), which overcomes the existing limitations. We propose the low-cost NTT and inverse-NTT converter that only use adders and shifters for on-chip domain transform, which solves the inflated bandwidth problem and enables more parallel computations in the accelerator. We also propose the accelerator architecture that includes multiple tile engines with the optimized data flow and mapping. Finally, we implement the proposed NTT-based CNN accelerator on the Xilinx Alveo U50 FPGA and evaluate it for popular deep CNN models. As a result, the proposed accelerator achieves 2859.5, 990.3, and 805.6 GOPS throughput for VGG-16, GoogLeNet, and Darknet-19, respectively. It outperforms the existing fast convolution-based CNN accelerators up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$9.6\times $ </tex-math></inline-formula> .

ARBiS: A Hardware-Efficient SRAM CIM CNN Accelerator With Cyclic-Shift Weight Duplication and Parasitic-Capacitance Charge Sharing for AI Edge Application

Computing-in-memory (CIM) relieves the Von Neumann bottleneck by storing the weights of neural networks in memory arrays. However, two challenges still exist, hindering the efficient acceleration of convolutional neural networks (CNN) in artificial intelligence (AI) edge devices. Firstly, the activations for sliding window (SW) operations in CNN still bring high memory access pressure. This can be alleviated by increasing the SW parallelism, but simple array replication suffers from poor array utilization and large peripheral circuits overhead. Secondly, the partial sums from individual CIM arrays, which are usually accumulated to obtain the final sum, introduce large latency due to enormous shift-and-add operations. Moreover, high-resolution ADCs are also needed to reduce the quantization error of partial sums, further increasing the hardware costs. In this paper, a hardware-efficient CIM accelerator, ARBiS, is proposed with improved activation reusability and bit-scalable matrix-vector-multiplication (MVM) for CNN acceleration in AI edge applications. The cyclic-shift weight duplication exploits a third dimension of receptive field (RF) depth for SW weight mapping to reduce the memory accesses of activations, improving the array utilization. The parasitic-capacitance charge sharing is employed to realize high-precision analog MVM in order to reduce the ADC cost. Compared with conventional architectures, ARBiS with parallel processing of 9 SW operations achieves 56.6%~58.8% alleviation of memory access pressure. Meanwhile, ARBiS configured with 8-bit ADCs saves 92.53%~94.53% ADC energy consumption. An ARBiS accelerator is evaluated to realize a computational efficiency (CE) of 10.28 (10.43) TOPS/mm2, an energy efficiency (EE) of 91.19 (112.36) TOPS/W with 8-bit (4-bit) ADCs, achieving <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$11.4\sim 11.7\times $ </tex-math></inline-formula> ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$11.6\sim 11.8\times $ </tex-math></inline-formula> ), <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.1\sim 3.3\times $ </tex-math></inline-formula> ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.4\sim 4\times $ </tex-math></inline-formula> ) improvements over state-of-the-art works, respectively.

Novel CNN Accelerator Design With Dual Benes Network Architecture

We presented a novel hardware architecture that uses dual Benes networks to accelerate Convolutional Neural Network (CNN) algorithms. This architecture can reduce the need for high-speed buses and maintain a high-speed connection between execution units and memories. Also, this design can work with multiple neural network models by changing the Benes network configuration only due to the reorderability of the non-blocking networks. The proposed architecture can save the time of re-implementing the hardware design. Thus, this architecture can act as a general CNN accelerator. We successfully implemented our CNN accelerator on the Xilinx Virtex UltraScale+ VU19P and evaluated it by running the CNN model LeNet-5.

A Low-Power 12-Bit SAR ADC for Analog Convolutional Kernel of Mixed-Signal CNN Accelerator

As deep learning techniques such as Convolutional Neural Networks (CNNs) are widely adopted, the complexity of CNNs is rapidly increasing due to the growing demand for CNN accelerator system-on-chip (SoC). Although conventional CNN accelerators can reduce the computational time of learning and inference tasks, they tend to occupy large chip areas due to many multiply-and-accumulate (MAC) operators when implemented in complex digital circuits, incurring excessive power consumption. To overcome these drawbacks, this work implements an analog convolutional filter consisting of an analog multiply-and-accumulate arithmetic circuit along with an analog-to-digital converter (ADC). This paper introduces the architecture of an analog convolutional kernel comprised of low-power ultra-small circuits for neural network accelerator chips. ADC is an essential component of the analog convolutional kernel used to convert the analog convolutional result to digital values to be stored in memory. This work presents the implementation of a highly low-power and area-efficient 12-bit Successive Approximation Register (SAR) ADC. Unlink most other SAR-ADCs with differential structure; the proposed ADC employs a single-ended capacitor array to support the preceding single-ended max-pooling circuit along with minimal power consumption. The SAR ADC implementation also introduces a unique circuit that reduces kick-back noise to increase performance. It was implemented in a test chip using a 55 nm CMOS process. It demonstrates that the proposed ADC reduces Kick-back noise by 40% and consequently improves the ADC’s resolution by about 10% while providing a near rail-to-rail dynamic range with significantly lower power consumption than conventional ADCs. The ADC test chip shows a chip size of 4600 μm² with a power consumption of 6.6 μW while providing an signal-to-noise-and-distortion ratio (SNDR) of 68.45 dB, corresponding to an effective number of bits (ENOB) of 11.07 bits.

A 1T2R1C ReRAM CIM Accelerator With Energy-Efficient Voltage Division and Capacitive Coupling for CNN Acceleration in AI Edge Applications

Resistive random-access memory (ReRAM) is widely studied in computing-in-memory (CIM) for neural network acceleration in edge devices. However, the static current of conventional 2T2R array becomes prominent with increasing computing parallelism, leading to remarkable power and area costs. To resolve this issue, a voltage-style one-transistor-two-resistor-one-capacitor ReRAM (1T2R1C) CIM accelerator using energy-efficient voltage division and capacitive coupling is proposed for convolutional neural network (CNN) acceleration in AI edge applications. The 1T2R1C cell, which is 15% smaller than the previous 2T2R cell, comprises one selection transistor, two ReRAM resistors for 1-bit weight storage, and a MOS-based capacitor. The multiply-and-accumulation (MAC) operation is realized by voltage division within a cell and capacitive coupling across different cells on a plate line. The static current during computation can be effectively reduced by the series connection of low-resistance state (LRS) and high-resistance state (HRS) ReRAM resistors within a cell and the elimination of low resistance paths in the array. A corresponding weight mapping method is also proposed to transform the multi-bit weight based on 0/1 into that on −1/1, adapting the proposed accelerator for high-precision CNN applications. By evaluation in the 28nm technology, the proposed accelerator with a 384Kb array achieves a peak energy efficiency of 400.2 TOPS/W, ~8.4X higher than previous work. The inference accuracy of CNN reaches 96.2% on the MNIST dataset.

Structured Term Pruning for Computational Efficient Neural Networks Inference

The state-of-the-art convolutional neural network accelerators are showing a growing interest in exploiting the bit-level sparsity and eliminating the ineffectual computations of zero bits. However, the excessive redundancy and the irregular distribution of nonzero bits limit the real speedup in the accelerators. To address this, we propose an algorithm-architecture codesign, named structured term pruning (STP), to boost the computation efficiency of neural networks inference. Specifically, we enhance the bit sparsity by guiding the weights toward the value with fewer power-of-two terms. Then, we structure the terms with layer-wise group budgets. Retraining is adopted to recover the accuracy drop. We also design the hardware of the group processing element and the fast signed-digital encoder for efficient implementation of STP networks. The system design of STP is realized with some easy alterations on an input stationary systolic array design. Extensive evaluation results demonstrate that STP can reduce significant inference computation costs, and achieve <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.35\times $ </tex-math></inline-formula> computational energy saving for the ResNet18 network on the ImageNet dataset.

Unconventional Integrated Photonic Accelerators for High-Throughput Convolutional Neural Networks

We provide an overview of the rapidly evolving landscape of integrated photonic neuromorphic architectures, specifically targeting the implementation of convolutional neural networks. The exploding research momentum stems from the well-known advantages of photonic circuits compared to digital electronics, and at the same time, it is driven by the massive need for cognitive image/video processing. In this context, we provide a detailed literature review on photonic cores operating as convolutional neural networks, covering either the functionality of a conventional neural network or its spiking counterpart. Moreover, we propose 2 alternative photonic approaches that refrain from simply transferring neural network concepts directly into the optical domain; instead, they focus on fusing photonic, digital electronic, and event-based bioinspired processing to optimally exploit the virtues of each scheme. These approaches can offer beyond state-of-the-art performance while relying on realistic, scalable technology. The first approach is based on a photonic integrated platform and a bioinspired spectrum-slicing technique. The photonic chip allows feature extraction through optical filtering with low power consumption and an equivalent computational efficiency of 72 femtojoules per multiply-and-accumulate operation for 5-bit precision. When combined with typical digital neural networks, an almost 5-fold reduction in the number of parameters was achieved with a minor loss of accuracy compared to established convolutional neural networks. The second approach follows a bioisomorphic route in which miniaturized spiking laser neurons and unsupervised bioinspired training are unified in a deep architecture, revealing a noise-resilient and power-efficient proposition.

ORSAS: An Output Row-Stationary Accelerator for Sparse Neural Networks

Various pruning skills and network compression methods make modern neural networks sparse for both weights and activations. However, GPUs (graphics processing units) and most of the customized CNN (convolutional neural networks) accelerators have not taken advantage of the sparsity of neural networks, and the accelerators for sparse neural networks in these years are suffering from low computational resource utilization. This paper first proposes an output row-stationary dataflow that exploits the sparsity of both weights and activations. It allows the accelerator to process weights and activations in their compressed form, leading to the high utilization of computational resources, i.e. multipliers. Besides, a low-cost compression algorithm is adopted for both weights and input activations to reduce the power consumption of data access. Secondly, the Y-buffer is proposed to eliminate repeated reading of input activations caused by halo effects, which emerge when tiling large-size input feature maps (ifmaps). Third, an interleaved broadcasting mechanism is introduced to alleviate the imbalance problems caused by the irregularity of sparse data. Finally, a prototype design called ORSAS is synthesized and placed and routed with a SMIC 55 nm process. The evaluation results show that ORSAS occupies the smallest logical cell area, and has the highest multiplier utilization among the peers’ works when the sparsity is large. ORSAS keeps the utilization of multipliers in a range from 60% to 90% over the convolutional layers of popular sparse CNNs and achieves ultra-low power consumption and highest efficiency.

Special Issue on Cool Chips

This special issue of IEEE Micro captures two contributions from among the 12 regular presentations of Cool Chips 25. The two articles are focused on a system-on- chip (SoC) with ternary neural network (TNN)/temporal convolutional neural network (TCN) accelerator, and body bias optimization for coarse-grained reconfigurable architectures (CGRAs), respectively. All of them are the emerging topics at Cool Chips 25.

A Reconfigurable CNN-Based Accelerator Design for Fast and Energy-Efficient Object Detection System on Mobile FPGA

In limited-resource edge computing circumstances such as on mobile devices, IoT devices, and electric vehicles, the energy-efficient optimized convolutional neural network (CNN) accelerator implemented on mobile Field Programmable Gate Array (FPGA) is becoming more attractive due to its high accuracy and scalability. In recent days, mobile FPGAs such as the Xilinx PYNQ-Z1/Z2 and Ultra96, definitely have the advantage of scalability and flexibility for the implementation of deep learning algorithm-based object detection applications. It is also suitable for battery-powered systems, especially for drones and electric vehicles, to achieve energy efficiency in terms of power consumption and size aspect. However, it has the low and limited performance to achieve real-time processing. In this article, optimizing the accelerator design flow in the register-transfer level (RTL) will be introduced to achieve fast programming speed by applying low-power techniques on FPGA accelerator implementation. In general, most accelerator optimization techniques are conducted on the system level on the FPGA. In this article, we propose the reconfigurable accelerator design for a CNN-based object detection system on the register-transfer level on mobile FPGA. Furthermore, we present RTL optimization design techniques that will be applied such as various types of clock gating techniques to eliminate residual signals and to deactivate the unnecessarily active block. Based on the analysis of the CNN-based object detection architecture, we analyze and classify the common computing operation components from the Convolutional Neuron Network, such as multipliers and adders. We implement a multiplier/adder unit to a universal computing unit and modularize it to be suitable for a hierarchical structure of RTL code. The proposed system design was tested with Resnet-20 which has 23 layers and it was trained with the dataset, CIFAR-10 which provides a test set of 10,000 images in several formats, and the weight data we used for this experiment was provided from Tensil. Experimental results show that the proposed design process improves the power efficient consumption, hardware utilization, and throughput by 16%, up to 58%, and 15%, respectively.

Design and implementation of convolution neural network accelerator for Winograd algorithm based on FPGA

Design and implementation of convolution neural network accelerator for Winograd algorithm based on FPGA

Photonic Convolutional Neural Network Accelerator Assisted by Phase Change Material

Photonic Convolutional Neural Network Accelerator Assisted by Phase Change Material

Integrated Imager and 3.22 μs/Kernel-Latency All-Digital In-Imager Global-Parallel Binary Convolutional Neural Network Accelerator for Image Processing

Integrated Imager and 3.22 <i>μ</i>s/Kernel-Latency All-Digital In-Imager Global-Parallel Binary Convolutional Neural Network Accelerator for Image Processing

Sustainable AI Processing at the Edge

Edge computing is a popular paradigm for accelerating light- to medium-weight machine learning algorithms initiated from mobile devices without requiring the long communication latencies to send them to remote datacenters in the cloud. Edge servers primarily consider traditional concerns, such as size, weight, and power constraints for their installations. However, such metrics are not entirely sufficient to consider environmental impacts from computing given the significant contributions from embodied energy and carbon. In this article we explore the tradeoffs of hardware strategies for convolutional neural network acceleration engines considering inference and online training. In particular, we explore the use of mobile graphics processing unit (GPU) accelerators, recently released edge-class field-programmable gate arrays, and novel processing in memory (PIM) using dynamic random-access memory (DRAM) and emerging Racetrack memory. Given edge servers already employ DRAM and sometimes GPU accelerators, we consider the sustainability implications using breakeven analysis of replacing or augmenting DDR3 with Racetrack memory. We also consider the implications for provisioning edge servers with different accelerators using indifference analysis. While mobile GPUs are typically much more energy efficient, their significant embodied energy can make them less sustainable than PIM solutions in certain scenarios that consider activity time and compute effort.

An Uninterrupted Processing Technique-Based High-Throughput and Energy-Efficient Hardware Accelerator for Convolutional Neural Networks

This article proposes an uninterrupted processing technique for the convolutional neural network (CNN) accelerator. It primarily allows the CNN accelerator to simultaneously perform both processing element (PE) operation and data fetching that reduces its latency and enhances the achievable throughput. Corresponding to the suggested technique, this work also presents a low latency VLSI-architecture of the CNN accelerator using the new random access line-buffer (RALB)-based design of PE array. Subsequently, the proposed CNN-accelerator architecture has been further optimized by reusing the local data in PE array, incurring better energy conservation. Our CNN accelerator has been hardware implemented on Zynq-UltraScale + MPSoC-ZCU102 FPGA board, and it operates at a maximum clock frequency of 340 MHz, consuming 4.11 W of total power. In addition, the suggested CNN accelerator with 864 PEs delivers a peak throughput of 587.52 GOPs and an adequate energy efficiency of 142.95 GOPs/W. Comparison of aforementioned implementation results with the literature has shown that our CNN accelerator delivers 33.42% higher throughput and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$6.24\times $ </tex-math></inline-formula> better energy efficiency than the state-of-the-art work. Eventually, the field-programmable gate array (FPGA) prototype of the proposed CNN accelerator has been functionally validated using the real-world test setup for the detection of object from input image, using the GoogLeNet neural network.

An Energy-Efficient Mixed-Bit CNN Accelerator With Column Parallel Readout for ReRAM-Based In-Memory Computing

Computing-In-memory (CIM) accelerators have the characteristics of storage and computing integration, which has the potential to break through the limit of Moore’s law and the bottleneck of Von-Neumann architecture for convolutional neural networks (CNN) implementation improvement. However, the performance of CIM accelerators is still limited by conventional CNN architectures and inefficient readouts. To increase energy-efficient performance, an optimized CNN model is required and a low-power column parallel readout is necessary for edge-computing hardware. In this work, an ReRAM-based CNN accelerator is designed. Mixed-bit operations from 1 bit to 8 bits are supported by an effective bitwidth configuration scheme to implement Neural Architecture Search (NAS)-optimized layer-wise multi-bit CNNs. Besides, column-parallel readout is achieved with excellent energy-efficient performance by a variation-reduction accumulation mechanism and low-power readout circuits. Additionally, we further explore systolic data reuse in an ReRAM-based PE array. Experiments are implemented on NAS-optimized ResNet-18. Benchmarks show that the proposed ReRAM accelerator can achieve peak energy efficiency of 2490.32 TOPS/W for 1-bit operation and average energy efficiency of 479.37 TOPS/W for <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1\sim 8$ </tex-math></inline-formula> -bit operations with evaluating NAS-optimized multi-bitwidth CNNs. When compared with the state-of-the-art works, the proposed accelerator shows at least <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$14.18{\times }$ </tex-math></inline-formula> improvement on energy efficiency.