DNN Accelerators Research Articles

Static Scheduling of Weight Programming for DNN Acceleration with Resource Constrained PIM

Most existing architectural studies on ReRAM-based processing-in-memory (PIM) DNN accelerators assume that all weights of the DNN can be mapped to the crossbar at once. However, these studies are over-idealized. ReRAM crossbar resources for calculation are limited because of technological limitations, so multiple weight mapping procedures are required during the inference process. In this article, we propose a static scheduling framework which generates the mapping between DNN weights and ReRAM cells with minimum runtime weight programming cost. We first build a ReRAM crossbar programming latency model by simultaneously considering the DNN weight patterns, ReRAM programming operations, and PIM architecture characteristics. Then, the model is used in the searching process to obtain an optimized weight-to-OU mapping table with minimum online programming latency. Finally, an OU scheduler is used to coordinate the activation sequences of OUs in the crossbars to perform the inference computation correctly. Evaluation results show the proposed framework significantly reduces the weight programming overhead and the overall inference latency for various DNN models with different input datasets.

FPGA-assisted Design Space Exploration of Parameterized AI Accelerators: A Quickloop Approach

FPGAs facilitate prototyping and debug, and recently accelerate full-stack simulations due to their rapid turnaround time (TAT). However, this TAT is restrictive in exhaustive design space explorations of parameterized RTL generators, especially DNN accelerators that unleash an explosive full-stack search space. This paper presents Quickloop, an efficient and scalable framework to enable FPGA-accelerated exploration. Quickloop first abstracts away the cumbersome flow of RTL generation, software stack, FPGA toolflow, workload execution and metrics extraction by wrapping these stages into isolated Quicksteps, featuring cascadability, scalability, and replay. Then, we analytically minimize the FPGA toolflow TAT via a novel, data-driven strategy that intelligently utilizes build fragments from previous iterations, enhancing the loop efficiency and simultaneously lowering the toolflow’s compute utilization.Quickloop is built around the OpenAI Gym environment framework and thus supports drop-in regression and reinforcement learning explorations. With a Quickloop around a reference Berkeley’s Gemmini DNN accelerator, we exhaustively explore its parameter space and discover complex performance patterns, based on full-stack simulation of Imagenet benchmarks as a workload. Compared to conventional FPGA toolflow, we further show that Quickloop effectively reduces episodal time by above 30%, as the episode approaches realistic lengths.

ALPRI-FI: A Framework for Early Assessment of Hardware Fault Resiliency of DNN Accelerators

ALPRI-FI: A Framework for Early Assessment of Hardware Fault Resiliency of DNN Accelerators

A Hybrid Sparse-dense Defensive DNN Accelerator Architecture against Adversarial Example Attacks

Understanding how to defend against adversarial attacks is crucial for ensuring the safety and reliability of these systems in real-world applications. Various adversarial defense methods are proposed, which aim at improving the robustness of neural networks against adversarial attacks by changing the model structure, adding detection networks, and adversarial purification network. However, deploying adversarial defense methods in existing DNN accelerators or defensive accelerators leads to many key issues. To address these challenges, this article proposes sDNNGuard , an elastic heterogeneous DNN accelerator architecture that can efficiently orchestrate the simultaneous execution of original ( target ) DNN networks and the detect algorithm or network. It not only supports for dense DNN detect algorithms, but also allows for sparse DNN defense methods and other mixed dense-sparse (e.g., dense-dense and sparse-dense) workloads to fully exploit the benefits of sparsity. sDNNGuard with a CPU core also supports the non-DNN computing and allows the special layer of the neural network, and used for the conversion for sparse storage format for weights and activation values. To reduce off-chip traffic and improve resources utilization, a new hardware abstraction with elastic on-chip buffer/computing resource management is proposed to achieve dynamical resource scheduling mechanism. We propose an extended AI instruction set for neural networks synchronization, task scheduling and efficient data interaction. Experiment results show that sDNNGuard can effectively validate the legitimacy of the input samples in parallel with the target DNN model, achieving an average 1.42× speedup compared with the state-of-the-art accelerators.

High-Performance Method and Architecture for Attention Computation in DNN Inference.

In recent years, The combination of Attention mechanism and deep learning has a wide range of applications in the field of medical imaging. However, due to its complex computational processes, existing hardware architectures have high resource consumption or low accuracy, and deploying them efficiently to DNN accelerators is a challenge. This paper proposes an online-programmable Attention hardware architecture based on compute-in-memory (CIM) marco, which reduces the complexity of Attention in hardware and improves integration density, energy efficiency, and calculation accuracy. First, the Attention computation process is decomposed into multiple cascaded combinatorial matrix operations to reduce the complexity of its implementation on the hardware side; second, in order to reduce the influence of the non-ideal characteristics of the hardware, an online-programmable CIM architecture is designed to improve calculation accuracy by dynamically adjusting the weights; and lastly, it is verified that the proposed Attention hardware architecture can be applied for the inference of deep neural networks through Spice simulation. Based on the 100nm CMOS process, compared with the traditional Attention hardware architectures, the integrated density and energy efficiency are increased by at least 91.38 times, and latency and computing efficiency are improved by about 12.5 times.

Investigation into designing VLSI of a flexible architecture for a deep neural network accelerator

Deep learning, a branch of AI, makes use of specialised neural networks. Computational acceleration of high-performance deep neural network algorithms continues to necessitate efficient architecture, high memory bandwidth, parallel processing, and resources, despite decades of study on such algorithms. Excessive space requirements are a problem for DNN implementations caused by resource-intensive components like activation functions (AF) and multiply-and-accumulate (MAC) units. In addition, edge-AI applications necessitate a densely packed, power-hungry, high-throughput DNN accelerator. If we want to build a DNN accelerator that uses little power and occupies little space, we need to optimise the MAC architecture, the AF, and the network complexity so that data flows efficiently. In addition, providing functional configurability while working with restricted chip surface is a difficulty for DNN hardware designs based on ASICs. This dissertation explores the efficient and low-power VLSI architecture of DNN accelerators, addressing the hardware implementation of DNN and targeting applications with limited resources. In order to assess MAC and non-linear AF operations, we investigate and enhance the CORDIC architecture. The poor throughput is a major downside of CORDIC-based architectures, even though they are area and power efficient. Consequently, we suggest a pipelined design for CORDIC-based MAC and AF that focuses on performance. Due to the increased hardware resource consumption that comes with pipeline stages, this study investigates the mutual exclusivity of CORDIC stages and investigates in depth the accuracy variation related to the number of stages needed to achieve high throughput.

INDM: Chiplet-Based Interconnect Network and Dataflow Mapping for DNN Accelerators

INDM: Chiplet-Based Interconnect Network and Dataflow Mapping for DNN Accelerators

3DNN-Xplorer: A Machine Learning Framework for Design Space Exploration of Heterogeneous 3-D DNN Accelerators

3DNN-Xplorer: A Machine Learning Framework for Design Space Exploration of Heterogeneous 3-D DNN Accelerators

High-Level Design of Precision-Scalable DNN Accelerators Based on Sum-Together Multipliers

High-Level Design of Precision-Scalable DNN Accelerators Based on Sum-Together Multipliers

Block-Wise Mixed-Precision Quantization: Enabling High Efficiency for Practical ReRAM-based DNN Accelerators

Block-Wise Mixed-Precision Quantization: Enabling High Efficiency for Practical ReRAM-based DNN Accelerators

MaGNAS: A Mapping-Aware Graph Neural Architecture Search Framework for Heterogeneous MPSoC Deployment

Graph Neural Networks (GNNs) are becoming increasingly popular for vision-based applications due to their intrinsic capacity in modeling structural and contextual relations between various parts of an image frame. On another front, the rising popularity of deep vision-based applications at the edge has been facilitated by the recent advancements in heterogeneous multi-processor Systems on Chips (MPSoCs) that enable inference under real-time, stringent execution requirements. By extension, GNNs employed for vision-based applications must adhere to the same execution requirements. Yet contrary to typical deep neural networks, the irregular flow of graph learning operations poses a challenge to running GNNs on such heterogeneous MPSoC platforms. In this paper, we propose a novel unified design-mapping approach for efficient processing of vision GNN workloads on heterogeneous MPSoC platforms. Particularly, we develop MaGNAS, a mapping-aware Graph Neural Architecture Search framework. MaGNAS proposes a GNN architectural design space coupled with prospective mapping options on a heterogeneous SoC to identify model architectures that maximize on-device resource efficiency. To achieve this, MaGNAS employs a two-tier evolutionary search to identify optimal GNNs and mapping pairings that yield the best performance trade-offs. Through designing a supernet derived from the recent Vision GNN (ViG) architecture, we conducted experiments on four (04) state-of-the-art vision datasets using both ( i ) a real hardware SoC platform (NVIDIA Xavier AGX) and ( ii ) a performance/cost model simulator for DNN accelerators. Our experimental results demonstrate that MaGNAS is able to provide 1.57 × latency speedup and is 3.38 × more energy-efficient for several vision datasets executed on the Xavier MPSoC vs. the GPU-only deployment while sustaining an average 0.11% accuracy reduction from the baseline.

A Survey of Design and Optimization for Systolic Array-based DNN Accelerators

In recent years, it has been witnessed that the systolic array is a successful architecture for DNN hardware accelerators. However, the design of systolic arrays also encountered many challenges. As DNN structures and applications become more complex, a DNN hardware accelerator based on the typical systolic array architecture suffers severe performance and efficiency penalties. So, it has motivated a significant amount of research on the redesign and optimization of the systolic array architecture. In this article, we survey these works on analyzing, redesigning, and improving the performance and efficiency of the systolic array architecture. These works are critical to the design flow of DNN accelerators based on systolic arrays. We also provide a technique classification of these works on the basis of their main research idea. Further, we attempt to compare the advantages and disadvantages of different designs and different technologies and provide quantitative results for reference. The aim of this survey is to provide researchers with knowledge of the state-of-the-art in the systolic array architecture and motivate them to design highly efficient DNN accelerators of tomorrow.

NeuroSpector: Systematic Optimization of Dataflow Scheduling in DNN Accelerators

NeuroSpector: Systematic Optimization of Dataflow Scheduling in DNN Accelerators

CD-MSA: Cooperative and Deadline-Aware Scheduling for Efficient Multi-Tenancy on DNN Accelerators

With DNN turning into the backbone of AI cloud services and propelling the emergence of INFerence-as-a-Service (INFaaS), DNN-specific accelerators have become the indispensable components of cloud inference systems. Due to the conservative “one-task-at-a-time” working mode and deadline blindness of those accelerators, implementing multi-tenancy that aims to improve the cost-effectiveness and meet SLA requirements is intractable. Recent studies including the temporal and spatial approaches, employ manifold scheduling mechanisms and sophisticated architecture innovations to address the challenge. However, these researches either still neglect the deadline awareness or render inevitable and expensive hardware overheads such as switches and storage. In this paper, we present <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Cooperative and Deadline-aware Multi-Systolic-Array scheduling</i> (CD-MSA), a low-cost solution for the cloud inference that utilizes the real time mechanism and task-level parallelism to enable efficient multi-tenancy. Based on our preemptive multi-systolic-array accelerator architecture supporting the simultaneous task co-location, we first construct a fine-grained DNN execution model to lay the groundwork for the lightweight preemption. Second, we design a cooperative, deadline- and laxity-aware scheduler in conjunction with an efficient schedulability test method for better QoS guarantee without introducing additional hardware cost. Finally, to further promote the overall throughput, we propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">dynamic task fusion</i> , a software approach that fuses different tasks into the logically “multi-threading” tasks at runtime. We compare CD-MSA with several state-of-the-art researches across three multi-DNN workloads. The evaluation results show CD-MSA improves the latency-bounded throughput, SLA satisfaction rate and weighted system throughput by up to 62%, 63% and 27%, respectively.

A Coordinated Model Pruning and Mapping Framework for RRAM-Based DNN Accelerators

Network sparsity or pruning is a pivotal technology for edge intelligence. RRAM-based accelerators, featuring dense storage and processing in memory capability, have demonstrated the superior computing performance and energy efficiency over the traditional CMOS-based accelerators for neural network applications. Unfortunately, RRAM-based accelerators suffer the performance or energy degradation when deploying pruned models, impairing their competition in the edge intelligence scenarios. We observed the essential reason is the pruning technology and the mapping strategy in prior RRAM-based accelerator and are optimized individually. As a result, the random zeros in the pruned DNN are irregularly distributed in the crossbars, rendering the degradation of computation parallelism of the crossbar without crossbar demand reduction. In this work, we propose a coordinated model pruning and mapping framework to jointly optimize of model accuracy and efficiency of RRAM-based accelerators. As for the mapping, we first decouple weight matrices in the bit-wise way and map the bit matrices to different crossbars, where the signed weights are represented with the two’s complement so as that save half desired crossbars. As for the pruning, we prune weight bits at the crossbar granularity so that free the crossbars holding the pruned bits. Furthermore, we employ an reinforcement learning (RL) approach to automatically select the optimal crossbar-aware bit-pruning strategy for any given neural network without laborious human efforts. We conducted the experiments on a set of representative neural networks and compared our framework with the state-of-the-art (SOTA) bit-sparsity works. The results show that ASBP saves up to 89.64% energy reduction and 84.12% area overhead compared to existing PRIME-like architecture. Besides, our framework outperforms the SOTA bit-sparsity design by 1.5x in terms of the energy reduction on the RRAM-based accelerator.

AdaPT: Fast Emulation of Approximate DNN Accelerators in PyTorch

Current state-of-the-art employs approximate multipliers to address the highly increased power demands of DNN accelerators. However, evaluating the accuracy of approximate DNNs is cumbersome due to the lack of adequate support for approximate arithmetic in DNN frameworks. We address this inefficiency by presenting AdaPT, a fast emulation framework that extends PyTorch to support approximate inference as well as approximation-aware retraining. AdaPT can be seamlessly deployed and is compatible with the most DNNs. We evaluate the framework on several DNN models and application fields including CNNs, LSTMs, and GANs for a number of approximate multipliers with distinct bitwidth values. The results show substantial error recovery from approximate re-training and reduced inference time up to 53.9x with respect to the baseline approximate implementation.

PRAP-PIM: A weight pattern reusing aware pruning method for ReRAM-based PIM DNN accelerators

PRAP-PIM: A weight pattern reusing aware pruning method for ReRAM-based PIM DNN accelerators

Designing a Performance-Centric MAC Unit with Pipelined Architecture for DNN Accelerators

Designing a Performance-Centric MAC Unit with Pipelined Architecture for DNN Accelerators

Exposing Reliability Degradation and Mitigation in Approximate DNNs Under Permanent Faults

Approximate computing is known for enhancing deep neural network accelerators' energy efficiency by introducing inexactness with a tolerable accuracy loss. However, small accuracy variations may increase the sensitivity of these accelerators towards undesired subtle disturbances, such as permanent faults. The impact of permanent faults in accurate deep neural network (AccDNN) accelerators has been thoroughly investigated in the literature. Conversely, the impact of permanent faults and their mitigation in approximate DNN (AxDNN) accelerators is vastly under-explored. Towards this, we first present an extensive fault resilience analysis of approximate multi-layer perceptrons (MLPs) and convolutional neural networks (CNNs) using the state-of-the-art Evoapprox8b multipliers in GPU and TPU accelerators. Then, we propose a novel fault mitigation method, i.e., fault-aware retuning of weights (Fal-reTune). Fal-reTune retunes the weights using a weight mapping function in the presence of faults for improved classification accuracy. To evaluate the fault resilience and the effectiveness of our proposed mitigation method, we used the most widely used MNIST, Fashion-MNIST, and CIFAR10 datasets. Our results demonstrate that the permanent faults exacerbate the accuracy loss in AxDNNs compared to the AccDNN accelerators. For instance, a permanent fault in AxDNNs can lead to 56\% accuracy loss, whereas the same faulty bit can lead to only 4\% accuracy loss in AccDNN accelerators. We empirically show that our proposed Fal-reTune mitigation method improves the performance of AxDNNs up to 98%, even with fault rates of up to 50%. Furthermore, we observe that the fault resilience in AxDNNs is orthogonal to their energy efficiency.

Targeting DNN Inference Via Efficient Utilization of Heterogeneous Precision DNN Accelerators

Modern applications rely more and more on the simultaneous execution of multiple DNNs, and Heterogeneous DNN Accelerators (HDAs) prevail as a solution to this trend. In this work, we propose, implement, and evaluate low precision Neural Processing Units (NPUs) which serve as building blocks to construct HDAs, to address the efficient deployment of multi-DNN workloads. Moreover, we design and evaluate HDA designs that increase the overall throughput, while reducing the energy consumption during NN inference. At the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">design time</i> , we implement HDAs inspired by the big.LITTLE computing paradigm, consisting of 8-bit NPUs together with lower precision bit-width NPUs. Additionally, an NN-to-NPU scheduling methodology is implemented to decide at <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">run-time</i> how to map the executed NN to the suitable NPU based on an accuracy drop threshold value. Our <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">hardware/software co-design</i> reduces the energy and response time of NNs by 29% and 10% respectively when compared to state-of-the-art homogeneous architectures. This comes with a negligible accuracy drop of merely 0.5%. Similar to the traditional CPU big.LITTLE, our asymmetric NPU design can open new doors for designing novel DNN accelerator architectures, due to their profound role in increasing the efficiency of DNNs with minimal losses in accuracy.