Neural Network Accelerator Research Articles

Realistic acceleration of neural networks with fine-grained tensor decomposition

As the modern deep neural networks (DNNs) have become more and more large-scale and expensive, the topic of DNN compression grows into a hot direction nowadays. Among variant compression methods, tensor decomposition seems to be the most promising and low-cost one because of its solid mathematical foundations and regular data structure. However, most of the existing tensor decompositions are not very good at accelerating DNNs, because there are always necessary transpositions on tensor modes to make the input data calculate with the decomposed factor tensors correctly, and transposition will bring extra memory and time cost for the realistic system without doubt. In this paper, we select a relatively novel Kronecker CANDECOMP/PARAFAC (KCP) tensor decomposition which has fine-grained factor tensors, and propose the transposition-free algorithm to calculate the contractions between the input data and the neural weight in KCP format. The theoretically analysis of computation complexity indicates that the proposed method is much more efficient than the existing algorithms. We further prove that the training complexity of KCP-DNN based on the proposed transposition-free algorithm can also be faster than the traditional ones, and make a comprehensive comparison of space and computation complexity including training and inference stages to show the superiority of our method. As a series of related works pay more attention to the recurrent neural networks (RNNs), we follow these existing practices and focus on the KCP-RNN to make a comprehensive comparison with them, and the experimental results show our KCP-RNN with transposition-free algorithm has systematically advantages including accuracy, space complexity, computation complexity, and realistic running time. Besides, some advanced characteristics of KCP-DNN such as collocation of ranks, have also been discussed.

A layer‐wise frequency scaling for a neural processing unit

A layer‐wise frequency scaling for a neural processing unit

A High Performance Multi-Bit-Width Booth Vector Systolic Accelerator for NAS Optimized Deep Learning Neural Networks

Multi-bit-width convolutional neural network (CNN) maintains the balance between network accuracy and hardware efficiency, thus enlightening a promising method for accurate yet energy-efficient edge computing. In this work, we develop state-of-the-art multi-bit-width accelerator for NAS Optimized deep learning neural networks. To efficiently process the multi-bit-width network inferencing, multi-level optimizations have been proposed. Firstly, differential Neural Architecture Search (NAS) method is adopted for the high accuracy multi-bit-width network generation. Secondly, hybrid Booth based multi-bit-width multiply-add-accumulation (MAC) unit is developed for data processing. Thirdly, vector systolic array is proposed for effectively accelerating the matrix multiplications. With vector-style systolic dataflow, both the processing time and logic resources consumption can be reduced when compared with the classical systolic array. Finally, The proposed multi-bit-width CNN acceleration scheme has been practically deployed on FPGA platform of Xilinx ZCU102. Average performance on accelerating the full NAS optimized VGG16 network is 784.2 GOPS, and peek performance of the convolutional layer can reach as high as 871.26 GOPS for INT8, 1676.96 GOPS for INT4, and 2863.29 GOPS for INT2 respectively, which is among the best results in previous CNN accelerator benchmarks.

Fast and High-Accuracy Approximate MAC Unit Design for CNN Computing

Multiply and accumulate (MAC) composed of a set of multipliers and one reduction dominates the latency and power of convolutional neural network (CNN) accelerators. Existing approximate multipliers reduce latency and power at a tolerable drop in accuracy, without considering the data distribution (implicitly assuming that data are uniformly distributed). This letter discloses that practical CNNs’ activations and weights are usually Gaussian-like distributed, and the bits of quantized activations and weights are typically not with a probability of 0.5. Thus, we propose an approximate MAC unit design by taking into account the statistical features of input data, to achieve a balanced tradeoff among latency, power, and accuracy. The extensive experiments show that our proposed MAC unit design provides much higher accuracy than state-of-the-art approximate circuits, while the latency, area, and power are similar.

An Efficient Deep Learning Accelerator Architecture for Compressed Video Analysis

Previous neural network accelerators tailored to video analysis only accept data of RGB/YUV domain, requiring decompressing the video that are often compressed before transmitted from the edge sensors. A compressed video processing accelerator can alleviate the decoding overhead, and gain performance speedup by operating on more compact input data. This work proposes a novel deep learning accelerator architecture, Alchemist, which is able to predict results directly from the compressed video bitstream instead of reconstructing the full RGB images. By utilizing the metadata of motion vector and critical blocks extracted from bitstreams, Alchemist contributes to a remarkable performance speedup of 5x with negligible accuracy loss. Nevertheless, we still find that the original compressed video coded by standard algorithms such as H.264 is not suitable to be directly manipulated, due to diverse compressed structures. Although obviating the requirement to recover all RGB frames, the accelerator must parse the entire compressed video bitstream to locate reference frames and extract useful metadata. If we combine the video codec with the proposed compressed video analysis, additional optimizations can be obtained. Therefore, to cope with the mismatch between current video coding algorithms, such as H.264 and neural network-based video analysis, we propose a specialized coding strategy to generate compressed video bitstreams more suitable for transmission and analysis, which further simplifies the decoding stage of video analysis and is capable of achieving significant storage reduction.

Chemical domain structure and its formation kinetics in CrCoNi medium-entropy alloy

The formation of local chemical order in medium-entropy alloys and high-entropy alloys (MEAs/HEAs) has been strongly suggested in recent experimental observations. Since chemical order can lead to changes in mechanical and functional properties, tailoring of chemical order is a promising approach for further improving those properties of MEAs and HEAs. However, details remain unclear regarding the atomic structure of the chemical order and the formation kinetics. Here, employing a large-scale Monte Carlo/molecular dynamics hybrid annealing simulation with a neural network potential, we find a chemical-domain structure (CDS) after annealing below 800 K in FCC CrCoNi MEA. In addition, the formation kinetics, such as the formation time and process and time–temperature–chemical-order diagrams of the CDS, were successfully obtained using a kinetic Monte Carlo simulation with artificial neural network acceleration. The findings provide key information for controlling chemical order via thermal processing.

A hardware-efficient computing engine for FPGA-based deep convolutional neural network accelerator

Deep convolutional neural networks (DCNNs) have recently emerged as a promising approach for computer vision tasks with many new DCNN architectures proposed to further improve their performance. However, the significant computation workload limits the deployment of such networks on embedded devices. Research on accelerating DCNN inference usually employs field-programmable gate arrays (FPGAs) due to their programmability. However, hardware efficiency and reconfigurability do not often receive sufficient attention. This paper proposes an efficient accelerator to support multiple DCNNs and improve the hardware utilization from three perspectives. First, a bandwidth-based tiling algorithm is used to improve the data transfer efficiency for direct memory access (DMA). Second, three parallel strategies are proposed to improve the utilization of the computing units (CUs). Third, a configurable CU is designed to improve the digital signal processor (DSP) utilization. The proposed accelerator is implemented on the Xilinx ZYNQ-7 ZC706 Evaluation Board at 200 MHz. The accelerator reaches 163 Giga Operations Per Second (GOPS) and 0.36 GOPS/DSP on the VGG-16 while consuming only 448 DSPs. A 0.24 GOPS/DSP is achieved with ResNet50 and 0.27 GOPS/DSP with YOLOv2-tiny. The experimental results demonstrate that this design achieves a better trade-off between hardware resource consumption, performance, and reconfigurability over previous works.

Sample-wise dynamic precision quantization for neural network acceleration

Sample-wise dynamic precision quantization for neural network acceleration

Built-In Functional Testing of Analog In-Memory Accelerators for Deep Neural Networks

The paper develops a methodology for the online built-in self-testing of deep neural network (DNN) accelerators to validate the correct operation with respect to their functional specifications. The DNN of interest is realized in the hardware to perform in-memory computing using non-volatile memory cells as computational units. Assuming a functional fault model, we develop methods to generate pseudorandom and structured test patterns to detect hardware faults. We also develop a test-sequencing strategy that combines these different classes of tests to achieve high fault coverage. The testing methodology is applied to a broad class of DNNs trained to classify images from the MNIST, Fashion-MNIST, and CIFAR-10 datasets. The goal is to expose hardware faults which may lead to the incorrect classification of images. We achieve an average fault coverage of 94% for these different architectures, some of which are large and complex.

A novel genetic algorithm-based approach for compression and acceleration of deep learning convolution neural network: an application in computer tomography lung cancer data

A novel genetic algorithm-based approach for compression and acceleration of deep learning convolution neural network: an application in computer tomography lung cancer data

A 65 nm Wireless Image SoC Supporting On-Chip DNN Optimization and Real-Time Computation-Communication Trade-Off via Actor-Critical Neuro-Controller

The widespread proliferation of smart sensors has led to hardware that enable edge intelligence (EI) with extreme energy efficiencies. This decreases the volume of data that is transmitted to the cloud, thus reducing: 1) processing latency; 2) communication energy; and 3) network congestion. However, this comes with an added cost of computation at the edge node. The cost (energy/latency) of edge computation and the cost of communication to the cloud vary widely depending on operating conditions, which include: 1) information content in the data; 2) algorithm selection; 3) channel conditions (noise, path-loss, etc.); 4) network size, available bandwidth; and 5) resources at the cloud. This article presents a 65 nm wireless image processing SoC for real-time computation-communication trade-off on resource-constrained edge devices. The test-chip includes: 1) an all-digital, near-memory, reconfigurable, and programmable neural-network (NN)-based systolic image processor; 2) a digitally adaptive radio-frequency digital-to-analog converter (RF-DAC)-based transceiver; and 3) a mixed-signal, time-based, actor-critic (AC) neuro-controller with compute-in-memory (CIM) and in-place weight updates that provide online learning and adaptation for efficiently controlling the computation, communication blocks separately as well as jointly. The major contributions of the proposed SoC are threefold: 1) a wireless Internet of Things (IoT) SoC architecture enabling a generic computation-communication trade-off scheme; 2) a novel CIM circuit design enabling effective AC control and online learning (0.59 pJ/MAC, 0.4 pJ/update); 3) integration of programmable deep NN (DNN) accelerator (1.05 TOPS/W) and reconfigurable transceiver (184 pJ/b @ –15 dBm) supporting versatile cloud-edge collaborations; and 4) significant system-level energy efficiency improvement (5.7 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times $ </tex-math></inline-formula> ) with real-time on-chip smart control enabled by seamless chip integration and AI-enabled decision-making. Furthermore, this SoC serves as a system-level IoT prototype for next-generation context-aware EI.

A 1Mb Mixed-Precision Quantized Encoder for Image Classification and Patch-Based Compression

Even if Application-Specific Integrated Circuits (ASIC) have proven to be a relevant choice for integrating inference at the edge, they are often limited in terms of applicability. In this paper, we demonstrate that an ASIC neural network accelerator dedicated to image processing can be applied to multiple tasks of different levels: image classification and compression, while requiring a very limited hardware. The key component is a reconfigurable, mixed-precision (3b/2b/1b) encoder that takes advantage of proper weight and activation quantizations combined with convolutional layer structural pruning to lower hardware-related constraints (memory and computing). We introduce an automatic adaptation of linear symmetric quantizer scaling factors to perform quantized levels equalization, aiming at stabilizing quinary and ternary weights training. In addition, a proposed layer-shared Bit-Shift Normalization significantly simplifies the implementation of the hardware-expensive Batch Normalization. For a specific configuration in which the encoder design only requires 1Mb, the classification accuracy reaches 87.5% on CIFAR-10. Besides, we also show that this quantized encoder can be used to compress image patch-by-patch while the reconstruction can performed remotely, by a dedicated full-frame decoder. This solution typically enables an end-to-end compression almost without any block artifacts, outperforming patch-based state-of-the-art techniques employing a patch-constant bitrate.

An Efficient CNN Accelerator for Low-Cost Edge Systems

Customized hardware based convolutional neural network ( CNN or ConvNet ) accelerators have attracted significant attention for applications in a low-cost, edge computing system. However, there is a lack of research that seeks to optimize at both the algorithm and hardware levels simultaneously in resource-constrained FPGA systems. In this paper, we first analyze ConvNet models to find one that is most suitable for a low-cost FPGA implementation. Based on the analysis, we select MobileNetV2 as the backbone of our research due to its hardware-friendly structure. We use a quantized implementation with 4-bit precision and optimize further with a smaller input resolution of 192 × 192 to obtain a 68.8% detection accuracy on ImageNet, which represents only a 3.2% accuracy loss compared to a floating-point model that uses the full input size. We then develop a hardware implementation that uses a low-cost FPGA. To accelerate the depth-wise separable ConvNet and utilize DRAM resources efficiently with parallel processing, we propose a novel scoreboard architecture to dynamically schedule DRAM data requests in order to maintain a high hardware utilization. The number of DSP blocks used is about six times smaller than in prior work. In addition, internal block RAM utilization is approximately nine times more efficient than in prior work. Our proposed design achieves 3.07 frames per second (FPS) on the low-cost and resource constrained FPGA system.

Meta-optic accelerators for object classifiers.

Rapid advances in deep learning have led to paradigm shifts in a number of fields, from medical image analysis to autonomous systems. These advances, however, have resulted in digital neural networks with large computational requirements, resulting in high energy consumption and limitations in real-time decision-making when computation resources are limited. Here, we demonstrate a meta-optic–based neural network accelerator that can off-load computationally expensive convolution operations into high-speed and low-power optics. In this architecture, metasurfaces enable both spatial multiplexing and additional information channels, such as polarization, in object classification. End-to-end design is used to co-optimize the optical and digital systems, resulting in a robust classifier that achieves 93.1% accurate classification of handwriting digits and 93.8% accuracy in classifying both the digit and its polarization state. This approach could enable compact, high-speed, and low-power image and information processing systems for a wide range of applications in machine vision and artificial intelligence.

CONNA: Configurable Matrix Multiplication Engine for Neural Network Acceleration

Convolutional neural networks (CNNs) have demonstrated promising results in various applications such as computer vision, speech recognition, and natural language processing. One of the key computations in many CNN applications is matrix multiplication, which accounts for a significant portion of computation. Therefore, hardware accelerators to effectively speed up the computation of matrix multiplication have been proposed, and several studies have attempted to design hardware accelerators to perform better matrix multiplications in terms of both speed and power consumption. Typically, accelerators with either a two-dimensional (2D) systolic array structure or a single instruction multiple data (SIMD) architecture are effective only when the input matrix has shapes that are close to or similar to a square. However, several CNN applications require multiplications of non-squared matrices with various shapes and dimensions, and such irregular shapes lead to poor utilization efficiency of the processing elements (PEs). This study proposes a configurable engine for neural network acceleration, called CONNA, whose computation engine can conduct matrix multiplications with highly utilized computing units, regardless of the access patterns, shapes, and dimensions of the input matrices by changing the shape of matrix multiplication conducted in the physical array. To verify the functionality of the CONNA accelerator, we implemented CONNA as an SoC platform that integrates a RISC-V MCU with CONNA on Xilinx VC707 FPGA. SqueezeNet on CONNA achieved an inference performance of 100 frames per second (FPS) with 2.36 mm2 and 83.55 mW in a 65 nm process, improving efficiency by up to 34.1 times better than existing accelerators in terms of FPS, silicon area, and power consumption.

Dataflow Optimization through Exploring Single-Layer and Inter-Layer Data Reuse in Memory-Constrained Accelerators

Off-chip memory access has become the performance and energy bottleneck in memory-constrained neural network accelerators. To provide a solution for the energy efficient processing of various neural network models, this paper proposes a dataflow optimization method for modern neural networks by exploring the opportunity of single-layer and inter-layer data reuse to minimize the amount of off-chip memory access in memory-constrained accelerators. A mathematical analysis of three inter-layer data reuse methods is first presented. Then, a comprehensive exploration to determine the optimal data reuse strategy from single-layer and inter-layer data reuse approaches is proposed. The result shows that when compared to the existing single-layer-based exploration method, SmartShuttle, the proposed approach can achieve up to 20.5% and 32.5% of off-chip memory access reduction for ResNeXt-50 and DenseNet-121, respectively.

CAP: Communication-Aware Automated Parallelization for Deep Learning Inference on CMP Architectures

Real-time inference of deep learning models on embedded and energy-efficient devices becomes increasingly desirable with the rapid growth of artificial intelligence on edge. Specifically, to achieve superb energy-efficiency and scalability, efficient parallelization of single-pass deep neural network (DNN) inference on chip multiprocessor (CMP) architectures is urgently required by many time-sensitive applications. However, as the number of processing cores scales up and the performance of cores has grown much fast, the on-chip inter-core data movement is prone to be a performance bottleneck for computation. To remedy this problem and further improve the performance of network inference, in this work, we introduce a communication-aware DNN parallelization technique called CAP, by exploiting the elasticity and noise-tolerance of deep learning algorithms on CMP. Moreover, in the hope that the conducted studies can provide new design values for real-time neural network inference on embedded chips, we also have evaluated the proposed approach on both multi-core Neural Network Accelerators (NNA) chips and general-purpose chip-multiprocessors. Our experimental results show that the proposed CAP can achieve 1.12×-1.65× system speedups and 1.14×-2.70× energy efficiency for different neural networks while maintaining the inference accuracy, compared to baseline approaches.

A Universal RRAM-Based DNN Accelerator With Programmable Crossbars Beyond MVM Operator

Resistive-RAM (RRAM)-based deep neural network (DNN) accelerator has shown a great potential as it is good at the matrix–vector multiplication (MVM) operator. However, it does not benefit non-MVM operators, such as transcendental activation or elementwise operations, which often require customized CMOS circuits in conventional DNN accelerator designs. In this article, we propose a new RRAM-based DNN inference accelerator, which leverages the proposed RRAM-CORDIC and RRAM-MLP algorithms to make the transcendental and elementwise operators calculable in the RRAM crossbar just like MVM. Both algorithms can exploit the higher multiply-and-accumulation (MAC) parallelism that is traditionally expensive in CMOS but now efficient in the RRAM crossbar. Then, we further propose an intercrossbar pipelining scheme, which can balance the number of crossbars for MVM and non-MVM operations and orchestrate them in pursuing higher DNN computing throughput. The experimental results show that both algorithms can sustain a high arithmetic accuracy and deliver less than 1% DNN accuracy loss on typical inference workloads. The elimination of expensive CMOS circuits, in turn, can trade more crossbar resources in the same area to speed up the performance by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.16\times $ </tex-math></inline-formula> to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.33\times $ </tex-math></inline-formula> . With the extended operators, the RRAM-based DNN accelerator can switch crossbar functions at will, and apply for a diverse of DNN models in a unified in-memory accelerator architecture.

Saving Energy of RRAM-Based Neural Accelerator Through State-Aware Computing

In-memory computing (IMC) is recognized as one of the most promising architecture solution to realize energy-efficient neural network inference. Amongst many memory technology, resistive RAM (RRAM) is a very attractive device to implement the IMC-based neural network accelerator architecture, which is particularly suitable for power-constrained IoT systems. Due to the nature of low leakage and in-situ computing, the dynamic power consumption of dot-production operations in RRAM crossbars dominates the chip power, especially when applied to low-precision neural networks. This work investigates the correlation between the cell resistance state and the crossbar operation power, and proposes a state-aware RRAM accelerator (SARA) architecture for energy-efficient low-precision neural networks. With the proposed state-aware network training and mapping strategy, crossbars in the RRAM accelerator can perform in a lower power state. Furthermore, we also leverage the proposed RRAM accelerator architecture to reduce the power consumption of high-precision network inference with both single-level or multilevel RRAM. The evaluation results show that for binary neural networks, our design saves 40.53% RRAM computing energy on average over the baseline. For high precision neural networks, the proposed method reduces 11.67% computing energy on average without any accuracy loss.

ComPreEND: Computation Pruning through Predictive Early Negative Detection for ReLU in a Deep Neural Network Accelerator

A vast amount of activation values of DNNs are zeros due to ReLU (Rectified Linear Unit), which is one of the most common activation functions used in modern neural networks. Since ReLU outputs zero for all negative inputs, the inputs to ReLU do not need to be determined exactly as long as they are negative. However, many accelerators usually do not consider such aspects of DNNs, losing a huge amount of opportunities for speedups and energy savings. To exploit such opportunities, we propose early negative detection (END), a computation pruning technique that detects the negative results at an early stage. The key to the early negative detection is the adoption of inverted two&#x0027;s complement representation for filter parameters. This ensures that as soon as the intermediate results become negative, the final results are guaranteed to be negative. Upon detection, the remaining computation can be skipped and the following ReLU output can be simply set to zero. We also propose a DNN accelerator architecture (ComPreEND) that takes advantage of such skipping. ComPreEND with END significantly improves both the energy efficiency and the performance according to the evaluation. Compared to the baseline, we obtain 20.5 and 29.3 percent speedup with accurate mode and predictive mode, and energy savings by 28.4 and 41.4 percent, respectively.