Deep Neural Network Accelerators Research Articles

Investigation into designing VLSI of a flexible architecture for a deep neural network accelerator

Deep learning, a branch of AI, makes use of specialised neural networks. Computational acceleration of high-performance deep neural network algorithms continues to necessitate efficient architecture, high memory bandwidth, parallel processing, and resources, despite decades of study on such algorithms. Excessive space requirements are a problem for DNN implementations caused by resource-intensive components like activation functions (AF) and multiply-and-accumulate (MAC) units. In addition, edge-AI applications necessitate a densely packed, power-hungry, high-throughput DNN accelerator. If we want to build a DNN accelerator that uses little power and occupies little space, we need to optimise the MAC architecture, the AF, and the network complexity so that data flows efficiently. In addition, providing functional configurability while working with restricted chip surface is a difficulty for DNN hardware designs based on ASICs. This dissertation explores the efficient and low-power VLSI architecture of DNN accelerators, addressing the hardware implementation of DNN and targeting applications with limited resources. In order to assess MAC and non-linear AF operations, we investigate and enhance the CORDIC architecture. The poor throughput is a major downside of CORDIC-based architectures, even though they are area and power efficient. Consequently, we suggest a pipelined design for CORDIC-based MAC and AF that focuses on performance. Due to the increased hardware resource consumption that comes with pipeline stages, this study investigates the mutual exclusivity of CORDIC stages and investigates in depth the accuracy variation related to the number of stages needed to achieve high throughput.

A tree-recursive partitioned multicast mechanism for NoC-based deep neural network accelerator

In chip multiprocessor systems (CMPs), Network on Chip (NoC) has been widely used due to its advantages of favorable reusability, high reliability, and low power consumption. Recently, using NoC platforms to accelerate deep neural networks (DNNs) has become a new trend. This design can enable the intermediate computation results of DNNs to be transmitted within the chip, reducing the number of accesses to off-chip memory. However, a large amount of one-to-many traffic in the DNN accelerator will occupy the system bandwidth, which will significantly reduce the performance of the NoC platform dominated by one-to-one traffic. To address this issue, we propose a tree-based recursive partitioning multicast scheme (TRPM), which increases the path diversity and improves the system bandwidth. We also design a single-cycle per-hop router architecture, which effectively enhances the transmission efficiency of multicast packets. Detailed simulation results show that compared with the latest tree-based multicast algorithm for DNN accelerators, our scheme reduces the number of routed packets by 35%, the classification latency by 13.5% and the average packet latency by 14.5% on average.

Fpar: filter pruning via attention and rank enhancement for deep convolutional neural networks acceleration

Fpar: filter pruning via attention and rank enhancement for deep convolutional neural networks acceleration

A Low-Latency DNN Accelerator Enabled by DFT-Based Convolution Execution Within Crossbar Arrays.

Analog resistive random access memory (RRAM) devices enable parallelized nonvolatile in-memory vector-matrix multiplications for neural networks eliminating the bottlenecks posed by von Neumann architecture. While using RRAMs improves the accelerator performance and enables their deployment at the edge, the high tuning time needed to update the RRAM conductance states adds significant burden and latency to real-time system training. In this article, we develop an in-memory discrete Fourier transform (DFT)-based convolution methodology to reduce system latency and input regeneration. By storing the static DFT/inverse DFT (IDFT) coefficients within the analog arrays, we keep digital computational operations using digital circuits to a minimum. By performing the convolution in reciprocal Fourier space, our approach minimizes connection weight updates, which significantly accelerates both neural network training and interference. Moreover, by minimizing RRAM conductance update frequency, we mitigate the endurance limitations of resistive nonvolatile memories. We show that by leveraging the symmetry and linearity of DFT/IDFTs, we can reduce the power by 1.57 × for convolution over conventional execution. The designed hardware-aware deep neural network (DNN) inference accelerator enhances the peak power efficiency by 28.02 × and area efficiency by 8.7 × over state-of-the-art accelerators. This article paves the way for ultrafast, low-power, compact hardware accelerators.

TECO: A Unified Feature Map Compression Framework Based on Transform and Entropy.

The massive memory accesses of feature maps (FMs) in deep neural network (DNN) processors lead to huge power consumption, which becomes a major energy bottleneck of DNN accelerators. In this article, we propose a unified framework named Transform and Entropy-based COmpression (TECO) scheme to efficiently compress FMs with various attributes in DNN inference. We explore, for the first time, the intrinsic unimodal distribution characteristic that widely exists in the frequency domain of various FMs. In addition, a well-optimized hardware-friendly coding scheme is designed, which fully utilizes this remarkable data distribution characteristic to encode and compress the frequency spectrum of different FMs. Furthermore, the information entropy theory is leveraged to develop a novel loss function for improving the compression ratio and to make a fast comparison among different compressors. Extensive experiments are performed on multiple tasks and demonstrate that the proposed TECO achieves compression ratios of 2.31 × in ResNet-50 on image classification, 3.47 × in UNet on dark image enhancement, and 3.18 × in Yolo-v4 on object detection while keeping the accuracy of these models. Compared with the upper limit of the compression ratio for original FMs, the proposed framework achieves the compression ratio improvement of 21%, 157%, and 152% on the above models.

Deep neural networks accelerators with focus on tensor processors

The massive amount of data and the problem of processing them is one of the main challenges of the digital age, and the development of artificial intelligence and machine learning can be useful in solving this problem. Using deep neural networks to improve the efficiency of these two areas is a good solution. So far, several architectures have been introduced for data processing with the benefit of deep neural networks, whose accuracy, efficiency, and computing power are different from each other. This article tries to review these architectures, their features, and their functions in a systematic way. According to the current research style, 24 articles (conference and research articles related to this topic) have been evaluated in the period of 2014–2022. In fact, the significant aspects of the selected articles are compared and at the end, the upcoming challenges and topics for future research are presented. The results show that the main parameters for proposing a new tensor processor include increasing speed and accuracy and reducing data processing time, reducing on-chip storage space, reducing DRAM access, reducing energy consumption, and achieving high efficiency.

Mapping of Deep Neural Network Accelerators on Wireless Multistage Interconnection NoCs

In the last few decades, the concept of Wireless Network-on-chip (WiNoC) has emerged as a promising alternative for Multiprocessor Systems on Chip (MPSOC) to achieve reliable and scalable communication. Worth recalling in this regard is that our research team has already designed, verified and evaluated Multistage Interconnection Networks (MIN) in this field. With respect to the present work, we consider proceeding with further exploring our thoughts on this research area. Firstly, we propose the design and performance evaluation of a hybrid (wireless/wired) MIN, analysing how this augmented network can potentially improve not only the average delay, but also energy consumption. Secondly, we continue with examining the implementation of our advanced DELTA-based MIN architecture on Deep Neural Network (DNN) accelerators, while accounting for its potential regularity and scalability in simultaneously maintaining an effective power efficiency and lower latency throughout the DNN operating process. In this context, several metrics have been evaluated in regard to three DNN application cases through implementation of their main respective modules.

Multiply-and-Fire: An Event-Driven Sparse Neural Network Accelerator

Deep neural network inference has become a vital workload for many systems from edge-based computing to data centers. To reduce the performance and power requirements for deep neural networks (DNNs) running on these systems, pruning is commonly used as a way to maintain most of the accuracy of the system while significantly reducing the workload requirements. Unfortunately, accelerators designed for unstructured pruning typically employ expensive methods to either determine non-zero activation-weight pairings or reorder computation. These methods require additional storage and memory accesses compared to the more regular data access patterns seen in structurally pruned models. However, even existing works that focus on the more regular access patterns seen in structured pruning continue to suffer from inefficient designs, which either ignore or expensively handle activation sparsity leading to low performance. To address these inefficiencies, we leverage structured pruning and propose the multiply-and-fire (MnF) technique, which aims to solve these problems in three ways: (a) the use of a novel event-driven dataflow that naturally exploits activation sparsity without complex, high-overhead logic; (b) an optimized dataflow takes an activation-centric approach, which aims to maximize the reuse of activation data in computation and ensures the data are only fetched once from off-chip global and on-chip local memory; and (c) based on the proposed event-driven dataflow, we develop an energy-efficient, high-performance sparsity-aware DNN accelerator. Our results show that our MnF accelerator achieves a significant improvement across a number of modern benchmarks and presents a new direction to enable highly efficient AI inference for both CNN and MLP workloads. Overall, this work achieves a geometric mean of 11.2× higher energy efficiency and 1.41× speedup compared to a state-of-the-art sparsity-aware accelerator.

TREAD-M3D: Temperature-Aware DNN Accelerators for Monolithic 3-D Mobile Systems

Monolithic 3D (MONO3D) integration provides performance and power efficiency benefits over 2D circuits and, thus, is a potent technology for the design of Deep Neural Network (DNN) accelerators with enhanced energy efficiency. However, high IC temperatures are major challenges for the design of MONO3D systems. To this end, this paper focuses on designing temperature-aware MONO3D DNN accelerators. We propose a new automated method, called TREAD-M3D, that provides a near-optimal MONO3D DNN accelerator architecture in terms of systolic array size, SRAM organization, partition across 3D layers, and operating frequency, for a given DNN, optimization goal, and temperature constraint. TREAD-M3D incorporates circuit-and architecture-level models to evaluate the power and performance characteristics of different partitions. Our method reveals valuable insights and enables tradeoff analysis for achieving high energy efficiency in MONO3D systolic arrays. In comparison to recent works that adopt a fixed partition choice to design MONO3D DNN systems, TREAD-M3D yields up to 22% higher energy efficiency. Using TREAD-M3D, we further demonstrate that temperature unawareness not only leads to infeasible configurations due to temperature violations but also over-estimates energy-delay-product benefits by up to 24%.

Joint Protection Scheme for Deep Neural Network Hardware Accelerators and Models

Deep neural networks (DNNs) are utilized in numerous image processing, object detection, and video analysis tasks and need to be implemented using hardware accelerators to achieve practical speed. Logic locking is one of the most popular methods for preventing chip counterfeiting. Nevertheless, existing logic-locking schemes need to sacrifice the number of input patterns leading to wrong output under incorrect keys to resist the powerful satisfiability (SAT)-attack. Furthermore, DNN model inference is fault-tolerant. Hence, using a wrong key for those SAT-resistant logic-locking schemes may not affect the accuracy of DNNs. This makes the previous SAT-resistant logic-locking scheme ineffective on protecting DNN accelerators. Besides, to prevent DNN models from being illegally used, the models need to be obfuscated by the designers before they are provided to end-users. Previous obfuscation methods either require long time to retrain the model or leak information about the model. This paper proposes a joint protection scheme for DNN hardware accelerators and models. The DNN accelerator is modified using a hardware key (Hkey) and a model key (Mkey). Different from previous logic locking, the Hkey, which is used to protect the accelerator, does not affect the output when it is wrong. As a result, the SAT attack can be effectively resisted. On the other hand, a wrong Hkey leads to substantial increase in memory accesses, inference time, and energy consumption and makes the accelerator unusable. A correct Mkey can recover the DNN model that is obfuscated by the proposed method. Compared to previous model obfuscation schemes, our proposed method avoids model retraining and does not leak model information.

A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification

Recent advancements in machine learning achieved by Deep Neural Networks (DNNs) have been significant. While demonstrating high accuracy, DNNs are associated with a huge number of parameters and computations, which leads to high memory usage and energy consumption. As a result, deploying DNNs on devices with constrained hardware resources poses significant challenges. To overcome this, various compression techniques have been widely employed to optimize DNN accelerators. A promising approach is quantization, in which the full-precision values are stored in low bit-width precision. Quantization not only reduces memory requirements but also replaces high-cost operations with low-cost ones. DNN quantization offers flexibility and efficiency in hardware design, making it a widely adopted technique in various methods. Since quantization has been extensively utilized in previous works, there is a need for an integrated report that provides an understanding, analysis, and comparison of different quantization approaches. Consequently, we present a comprehensive survey of quantization concepts and methods, with a focus on image classification. We describe clustering-based quantization methods and explore the use of a scale factor parameter for approximating full-precision values. Moreover, we thoroughly review the training of a quantized DNN, including the use of a straight-through estimator and quantization regularization. We explain the replacement of floating-point operations with low-cost bitwise operations in a quantized DNN and the sensitivity of different layers in quantization. Furthermore, we highlight the evaluation metrics for quantization methods and important benchmarks in the image classification task. We also present the accuracy of the state-of-the-art methods on CIFAR-10 and ImageNet. This article attempts to make the readers familiar with the basic and advanced concepts of quantization, introduce important works in DNN quantization, and highlight challenges for future research in this field.

AVATAR: An Aging- and Variation-Aware Dynamic Timing Analyzer for Error-Efficient Computing

As the timing guardband consumes more and more design margin with the technology scaling, better-than-worst-case (BTWC) techniques have gained more attention as a promising solution. BTWC techniques can relax the design margin by transcending the pessimistic static timing constraints and utilizing the dynamic timing information. However, to guarantee the design reliability throughout the lifetime, the conventional dynamic timing analysis (DTA) engines need an extra reliability guardband, which is commonly evaluated under the worst-case corners of aging and variation. This type of guardbanding consumes the precious design margin, thus hindering the efficiency improvement from BTWC techniques. Therefore, in this paper, we propose AVATAR, an aging-and variation-aware dynamic timing analyzer that can perform DTA with the impact of transistor aging and random process variation, including the gate-level aging analysis and random variation model that can accurately calculate cell delay under the impact of transistor aging and random variation, and an event-based DTA algorithm that avoids the pessimistic property of graph-based analysis. We also propose an ML-assisted DTA acceleration flow for the multicycle DTA of homogeneous multicore designs. We present two case studies using AVATAR to show its effectiveness. First, we present an application-based dynamic-voltage-frequency-scaling (DVFS) design methodology based on AVATAR, which can exploit application-level dynamic timing slack (DTS) to improve energy efficiency and performance. The results demonstrate that, compared to the design based on the conventional corner-based DTA, the additional performance improvement of the design based on AVATAR can be up to 14% or the additional power-saving can be up to 20%. Second, we demonstrate using the proposed ML-assisted acceleration flow for reliability-aware deep neural network (DNN) accelerator simulation. We use the proposed flow to estimate the impact of timing errors due to aging and random variation on the inference accuracy of two benchmark DNNs. The results demonstrate that the proposed acceleration flow achieves up to 10W speedup with an average error of less than 2%.

MGen: A Framework for Energy-Efficient In-ReRAM Acceleration of Multi-Task BERT

Recently, multiple transformer models, such as BERT, have been utilized together to support multiple natural language processing (NLP) tasks in a system, also known as multi-task BERT. Multi-task BERT with very high weight parameters increases the area requirement of a processing in resistive memory (ReRAM) architecture, and several works have attempted to address this model size issue. Despite the reduced parameters, the number of multi-task BERT computations remains the same, leading to massive energy consumption in ReRAM-based deep neural network (DNN) accelerators. Therefore, we suggest a framework for better energy efficiency during the ReRAM acceleration of multi-task BERT. First, we analyze the inherent redundancies of multi-task BERT and the computational properties of the ReRAM-based DNN accelerator, after which we propose what is termed the model generator, which produces optimal BERT models supporting multiple tasks. The model generator reduces multi-task BERT computations while maintaining the algorithmic performance. Furthermore, we present task scheduler, which adjusts the execution order of multiple tasks, to run the produced models efficiently. As a result, the proposed framework achieves maximally 4.4× higher energy efficiency over the baseline, and it can also be combined with the previous multi-task BERT works to achieve both a smaller area and higher energy efficiency.

OECS: a deep convolutional neural network accelerator based on a 3D hybrid optical-electrical NoC

Various emerging technologies have been proposed and applied to deep convolutional neural networks (DCNNs) to enhance their execution efficiency and accuracy while reducing the number of parameters. However, previous dataflow-based DCNN accelerators have primarily focused on accelerating the convolution (CONV) layer while reducing the overhead of data movement through specific data reuse strategies to improve performance. When deploying these emerging DCNNs in conventional accelerators, frequent data access from both the external memory and internal processing units of the accelerator can result in significant data movement overhead, leading to decreased acceleration performance. To address these challenges, this paper proposes a novel three-dimensional hybrid optical-electrical network-on-chip (NoC) accelerator called the optical-electronic channel-stationary system (OECS). OECS leverages a channel stationary (CS) calculation mode and stay-at-local data storage strategy to minimize the movement and processing of all data in the local processing element (PE). This strategy reduces data movement between each PE in the accelerator and between the accelerator and external memory, resulting in improved acceleration performance. Simulation results demonstrate that, compared to state-of-the-art accelerators, OECS achieves a 53% improvement in execution speed and saves approximately 44% of energy consumption associated with data movement.

ETA: An Efficient Training Accelerator for DNNs Based on Hardware-Algorithm Co-Optimization.

Recently, the efficient training of deep neural networks (DNNs) on resource-constrained platforms has attracted increasing attention for protecting user privacy. However, it is still a severe challenge since the DNN training involves intensive computations and a large amount of data access. To deal with these issues, in this work, we implement an efficient training accelerator (ETA) on field-programmable gate array (FPGA) by adopting a hardware-algorithm co-optimization approach. A novel training scheme is proposed to effectively train DNNs using 8-bit precision with arbitrary batch sizes, in which a compact but powerful data format and a hardware-oriented normalization layer are introduced. Thus the computational complexity and memory accesses are significantly reduced. In the ETA, a reconfigurable processing element (PE) is designed to support various computational patterns during training while avoiding redundant calculations from nonunit-stride convolutional layers. With a flexible network-on-chip (NoC) and a hierarchical PE array, computational parallelism and data reuse can be fully exploited, and memory accesses are further reduced. In addition, a unified computing core is developed to execute auxiliary layers such as normalization and weight update (WU), which works in a time-multiplexed manner and consumes only a small amount of hardware resources. The experiments show that our training scheme achieves the state-of-the-art accuracy across multiple models, including CIFAR-VGG16, CIFAR-ResNet20, CIFAR-InceptionV3, ResNet18, and ResNet50. Evaluated on three networks (CIFAR-VGG16, CIFAR-ResNet20, and ResNet18), our ETA on Xilinx VC709 FPGA achieves 610.98, 658.64, and 811.24 GOPS in terms of throughput, respectively. Compared with the prior art, our design demonstrates a speedup of 3.65x and an energy efficiency improvement of 8.54x on CIFAR-ResNet20.

STIFT: A Spatio-Temporal Integrated Folding Tree for Efficient Reductions in Flexible DNN Accelerators

Increasing deployment of Deep Neural Networks (DNNs) recently fueled interest in the development of specific accelerator architectures capable of meeting their stringent performance and energy consumption requirements. DNN accelerators can be organized around three separate NoCs, namely distribution, multiplier, and reduction networks (or DN, MN, and RN, respectively) between the global buffer(s) and the compute units (multipliers/adders). Among them, the RN, used to generate and reduce the partial sums produced during DNN processing, is a first-order driver of the area and energy efficiency of the accelerator. RNs can be orchestrated to exploit a Temporal, Spatial or Spatio-Temporal reduction dataflow. Among these, Spatio-Temporal reduction is the one that has shown superior performance. However, as we demonstrate in this work, a state-of-the-art implementation of the Spatio-Temporal reduction dataflow, based on the addition of Accumulators (Ac) to the RN (i.e., RN+Ac strategy), can result into significant area and energy expenses. To cope with this important issue, we propose STIFT (that stands for Spatio-Temporal Integrated Folding Tree ) that implements the Spatio-Temporal reduction dataflow entirely on the RN hardware substrate (i.e., without the need for the extra accumulators). STIFT results into significant area and power savings regarding the more complex RN+Ac strategy, at the same time its performance advantage is preserved.

An Electro-Photonic System for Accelerating Deep Neural Networks

The number of parameters in deep neural networks (DNNs) is scaling at about 5× the rate of Moore’s Law. To sustain this growth, photonic computing is a promising avenue, as it enables higher throughput in dominant general matrix-matrix multiplication (GEMM) operations in DNNs than their electrical counterpart. However, purely photonic systems face several challenges including lack of photonic memory and accumulation of noise. In this article, we present an electro-photonic accelerator, ADEPT, which leverages a photonic computing unit for performing GEMM operations, a vectorized digital electronic application-specific integrated circuits for performing non-GEMM operations, and SRAM arrays for storing DNN parameters and activations. In contrast to prior works in photonic DNN accelerators, we adopt a system-level perspective and show that the gains while large are tempered relative to prior expectations. Our goal is to encourage architects to explore photonic technology in a more pragmatic way considering the system as a whole to understand its general applicability in accelerating today’s DNNs. Our evaluation shows that ADEPT can provide, on average, 5.73× higher throughput per watt compared to the traditional systolic arrays in a full-system, and at least 6.8× and 2.5× better throughput per watt, compared to state-of-the-art electronic and photonic accelerators, respectively.

FlexBlock: A Flexible DNN Training Accelerator With Multi-Mode Block Floating Point Support

When training deep neural networks (DNNs), expensive floating point arithmetic units are used in GPUs or custom neural processing units (NPUs). To reduce the burden of floating point arithmetic, community has started exploring the use of more efficient data representations, e.g., block floating point (BFP). The BFP format allows a group of values to share an exponent, which effectively reduces the memory footprint and enables cheaper fixed point arithmetic for multiply-accumulate (MAC) operations. However, existing BFP-based DNN accelerators are targeted for a specific precision, making them less versatile. In this paper, we present FlexBlock, a DNN training accelerator with three BFP modes, possibly different among activation, weight, and gradient tensors. By configuring FlexBlock to a lower BFP precision, the number of MACs handled by the core increases by up to 4× in 8-bit mode or 16× in 4-bit mode compared to 16-bit mode. To reach this theoretical upper bound, FlexBlock maximizes the core utilization at various precision levels or layer types, and allows dynamic precision control to keep throughput at its peak without sacrificing training accuracy. We evaluate the effectiveness of FlexBlock using representative DNNs on CIFAR, ImageNet and WMT14 datasets. As a result, training in FlexBlock significantly improves training speed by 1.5 5.3× and energy efficiency by 2.4 7.0× compared to other training accelerators.

W-AMA: Weight-aware Approximate Multiplication Architecture for neural processing

This paper presents the Weight-aware Approximate Multiplication Architecture (W-AMA) for Deep Neural Networks (DNNs). Considering the Gaussian-like weight distribution, it deploys an accuracy-configurable computing component to improve the computational efficiency. Two techniques for effectively integrating the W-AMA into DNN accelerator are presented: (1) A Cartesian Genetic Programming (CGP) based approximate multiplier is designed and selectable to compute the Least Significant Bit (LSB) for a higher accuracy mode. The Reward–Penalty-Coefficient (RPC) is proposed to achieve the internal-compensation. (2) The Hessian-Aware-Approximation (HAA) method is utilized for hybrid approximate modes cross-layer mapping. Based on the W-AMA, an energy-efficient DNN accelerator is proposed and evaluated on 28 nm technology. It can achieve the energy efficiency of 9.6 TOPS/W, and the computational energy efficiency can be improved by 1.5× compared with the standard units, with an 0.52% accuracy loss on CIFAR-10 using ResNet-18.

A Low-Power DNN Accelerator Enabled by a Novel Staircase RRAM Array.

Enhancing the ubiquitous sensors and connected devices with computational abilities to realize visions of the Internet of Things (IoT) requires the development of robust, compact, and low-power deep neural network accelerators. Analog in-memory matrix-matrix multiplications enabled by emerging memories can significantly reduce the accelerator energy budget while resulting in compact accelerators. In this article, we design a hardware-aware deep neural network (DNN) accelerator that combines a planar-staircase resistive random access memory (RRAM) array with a variation-tolerant in-memory compute methodology to enhance the peak power efficiency by 5.64× and area efficiency by 4.7× over state-of-the-art DNN accelerators. Pulse application at the bottom electrodes of the staircase array generates a concurrent input shift, which eliminates the input unfolding, and regeneration required for convolution execution within typical crossbar arrays. Our in-memory compute method operates in charge domain and facilitates high-accuracy floating-point computations with low RRAM states, device requirement. This work provides a path toward fast hardware accelerators that use low power and low area.