Convolutional Neural Network Accelerator Research Articles

A ReRAM-Based Convolutional Neural Network Accelerator Using the Analog Layer Normalization Technique

This paper presents a ReRAM-based convolutional neural network (CNN) accelerator with a new analog layer normalization (ALN) technique. The proposed ALN can be used to effectively reduce the effect of the conductance variation is ReRAM devices by normalizing the outputs of the vector-matrix multiplication (VMM) in the charge domain. The ALN achieves high energy and hardware efficiencies because it directly processes normalization of the VMM outputs without storing their values in memory and is merged into the neuron circuit of the accelerator. To verify the effect of the ALN through experiments, a VMM accelerator that consists of two 25 × 25 sized ReRAM arrays and peripheral circuits with ALN is used for a convolution layer with digital signal processing (DSP) in a field programmable gate array (FPGA). The MNIST dataset is used to train and inference a CNN employing two VMM accelerators that work as convolution layers in a pipelined manner. Despite the conductance variation of the ReRAM devices, the ALN successfully stabilizes the output distribution of the convolution layer, which improves the classification accuracy of the network. A final classification accuracy for the MNIST and Fashion-MNIST datasets of 96.2% and 83.1% is achieved, respectively, with an energy efficiency of 9.94 tera-operations per second per Watt (TOPS/W).

Multiplication Circuit Architecture for Error- Tolerant CNN-Based Keywords Speech Recognition

Approximate multiplication is an emerging circuit design technique for AI-based IoT devices which can reduce the energy consumption. A precision reconfigurable tensor multiplication unit (TMU) is proposed to explore the advantages for energy efficient approximate computing for error-tolerant convolutional neural network (CNN) based keywords speech recognition, which is a widely used human-machine interaction system. A positive-negative encoding method and a reorganization structure of partial products are proposed for the tensor multiplication. The partial products are further optimized by a hierarchical addition tree structure with fine-grained precision reconfigurable approximate computing. The imprecision parts of the addition tree are optimized with a low supply voltage. The effects of circuit configuration variations on accuracy and power consumption have been evaluated under 22nm process technology. The proposed approximate designs are applied to a CNN-based keywords speech recognition system. With the proposed approach, the energy efficiency of the CNN accelerator can be greatly improved with a power reduction of up to 44.1% at a loss in accuracy of less than 2%.

Review on FPGA-based accelerators in convolutional neural network

In current research, an important means to realize artificial intelligence is artificial neural network. Many tedious problemscan be solved by artificial neural network, such as image classification, speech recognition, natural language processing. Among neural networks, although convolutional neural network has significantly better performance in the field of image recognition, it requires many parameters and a large amount of computation, and the number of network layers is gradually increasing with the progress of the algorithm, which leads to model sizes are getting larger and more difficult to handle, which has more stringent requirements on various aspects of hardware capabilities, such as computing power, data storage and memory bandwidth. Therefore, the research of acceleration is particularly important, especially in the field of hardware acceleration. And as an emerging hardware platform, field programmable gate array has the characteristics of rapid customization and programmability, which can greatly improve the efficiency of accelerators design, implementation, and verification. As a result, it can be widely used in the hardware-accelerated design of neural networks. Although field programmable gate array itself has some defects, many research results have proved its huge advantages and development potential. This paper mainly summarizes the acceleration of convolutional neural network based on field programmable gate array platform, and discusses its characteristics and application space by comparing with other platforms, showing its high applicability to convolutional neural network, and finally looking forward to the progress of related research work.

Convolutional Neural Networks using FPGA-based Pipelining

In order to speed up convolutional neural networks (CNNs), this study gives a complete overview of the use of FPGA-based pipelining for hardware acceleration of CNNs. These days, most people use convolutional neural networks (CNNs) to perform computer vision tasks like picture categorization and object recognition. The processing and memory demands of CNNs, however, can be excessive, especially for real-time applications. In order to speed up CNNs, FPGA-based pipelining has emerged as a viable option thanks to its parallel processing capabilities and low power consumption. The examination describes the fundamentals of FPGA-based pipelining and the basic structure of convolutional neural networks (CNNs). The current best practises for developing pipelined accelerators for CNNs on FPGAs are then reviewed, covering topics like partitioning and pipelining. Area and power limits, memory needs, and latency considerations are only some of the difficulties and trade-offs discussed in the article. In addition, the survey evaluates and contrasts the various pipelined FPGA accelerators for CNNs in terms of performance, energy consumption, and resource utilisation. Future directions and potential research areas are also discussed in the paper, such as the use of approximate computing techniques, the integration of reconfigurable architectures with emerging memory technologies, and the exploration of hybrid architectures that combine FPGAs and other hardware accelerators. This survey was created to aid researchers and practitioners in developing efficient and effective hardware accelerators for neural networks by providing a thorough overview of current trends and issues in FPGA-based pipelining for CNNs.

SoC-based real-time SVM classification with integrated training using HLS and PYNQ

Support Vector Machines (SVM) are widely used techniques in the field of classification problems because of their ability to effectively deal with datasets that have complex non-linear structures and a high dimensionality. The compute-intensive training algorithm associated with SVM makes it challenging to keep an up-to-date model that accurately reflects the characteristics of newly arriving data points in real-time systems. This paper proposes a novel training algorithm for incremental learning from large datasets, based on a variant of Sequential Minimal Optimization (SMO). High-Level Synthesis (HLS) was used for implementing the Field Programmable Gate Array (FPGA) based Intellectual Property (IP) Core, which includes the computationally intensive kernel computation portion of the training algorithm. In addition to the kernel computation, the inference phase of the SVM classifier is built into the IP core, and its use can be switched on the fly. The computational latency and memory bandwidth of an IP core are optimized using loop pipelining and DMA burst data transfer. With the help of hardware/software co-design, the IP core is integrated into the design of a flexible and re-usable System on Chip (SoC) called PYNQ Overlay. The experiments show that the overlay outperforms the embedded processor, multiple hardware SVM classifiers, and hardware accelerated Convolutional Neural Networks (CNN) in terms of real-time efficiency. The Overlay makes much less use of the resources available on the chip in comparison to the majority of the CNN accelerators. The overlay achieves an average classification accuracy that is only 1% lower than that of an ARM Cortex-A9 processor, according to experimental results on six datasets. Furthermore, it can increase training speed by an average of 31.82x and inference speed by an average of 31.74x. In addition, the proposed Overlay design achieves a 2.3x improvement in average training speed, as measured in Mega bits per second, compared to existing SVM training implementations, along with incremental learning and multi-class classification support.

X-NVDLA: Runtime Accuracy Configurable NVDLA Based on Applying Voltage Overscaling to Computing and Memory Units

This paper investigates a runtime accuracy reconfigurable implementation of an energy efficient deep learning accelerator. It is based on voltage overscaling (VOS) technique which provides dynamic adjustment of approximation level as well as improving lifetime/reliability of the accelerator. The technique is applied to both computing and memory units where based on the minimum required accuracy, the applied voltage is adjusted during the runtime. The implementation of the network is performed using NVDLA which is an open-source CNN (Convolutional Neural Network) accelerator. The approximation is applied to both the accelerator MAC array employed for the required network computations and to the accelerator SRAM memory utilized for storing the inputs (images), weights, and activation data. To control the accuracy degradation of the approximate accelerator (called X-NVDLA), the reduced voltage is applied only to LSB bits of the MAC array and SRAM unit. To assess the efficacy of the proposed energy efficient accelerator, the energy-accuracy characteristics of X-NVDLA when running LeNet-5 and ResNet-50 networks with 8-bit (integer) precision are investigated. In addition, the characteristic of bias temperature instability (BTI), as one of the lifetime deteriorating phenomena is determined. The study includes energy improvement versus accuracy degradation as a function of overscaled voltages and number of approximate the least significant bits using a 15nm FinFET technology.

CODEBench: A Neural Architecture and Hardware Accelerator Co-Design Framework

Recently, automated co-design of machine learning (ML) models and accelerator architectures has attracted significant attention from both the industry and academia. However, most co-design frameworks either explore a limited search space or employ suboptimal exploration techniques for simultaneous design decision investigations of the ML model and the accelerator. Furthermore, training the ML model and simulating the accelerator performance is computationally expensive. To address these limitations, this work proposes a novel neural architecture and hardware accelerator co-design framework, called CODEBench. It comprises two new benchmarking sub-frameworks, CNNBench and AccelBench, which explore expanded design spaces of convolutional neural networks (CNNs) and CNN accelerators. CNNBench leverages an advanced search technique, Bayesian Optimization using Second-order Gradients and Heteroscedastic Surrogate Model for Neural Architecture Search, to efficiently train a neural heteroscedastic surrogate model to converge to an optimal CNN architecture by employing second-order gradients. AccelBench performs cycle-accurate simulations for diverse accelerator architectures in a vast design space. With the proposed co-design method, called Bayesian Optimization using Second-order Gradients and Heteroscedastic Surrogate Model for Co-Design of CNNs and Accelerators, our best CNN–accelerator pair achieves 1.4% higher accuracy on the CIFAR-10 dataset compared to the state-of-the-art pair while enabling 59.1% lower latency and 60.8% lower energy consumption. On the ImageNet dataset, it achieves 3.7% higher Top1 accuracy at 43.8% lower latency and 11.2% lower energy consumption. CODEBench outperforms the state-of-the-art framework, i.e., Auto-NBA, by achieving 1.5% higher accuracy and 34.7× higher throughput while enabling 11.0× lower energy-delay product and 4.0× lower chip area on CIFAR-10.

Design of a Low-Power Super-Resolution Architecture for Virtual Reality Wearable Devices

Head-mounted displays (HMDs) have made a virtual reality (VR) accessible to a widespread consumer market, introducing a revolution in many applications. Among the limitations of current HMD technology, the need for generating high-resolution images and streaming them at adequate frame rates is one of the most critical. Super-resolution (SR) convolutional neural networks (CNNs) can be exploited to alleviate timing and bandwidth bottlenecks of video streaming by reconstructing high-resolution images locally (i.e., near the display). However, such techniques involve a significant amount of computations that makes their deployment within area-/power-constrained wearable devices often unfeasible. This research work originated from the consideration that the human eye can capture details with high acuity only within a certain region, called the fovea. Therefore, we designed a custom hardware architecture able to reconstruct high-resolution images by treating foveal region (FR) and peripheral region (PR) through accurate and inaccurate operations, respectively. Hardware experiments demonstrate the effectiveness of our proposal: a customized fast SR CNN (FSRCNN) accelerator realized as described here and implemented on a 28-nm process technology is able to process up to 214 ultrahigh definition frames/s, while consuming just 0.51 pJ/pixel without compromising the perceptual visual quality, thus achieving a 55% energy reduction and a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times 14$ </tex-math></inline-formula> times higher throughput rate, with respect to state-of-the-art competitors.

Sparse-Aware Deep Learning Accelerator

In view of the difficulty of hardware implementation of convolutional neural network computing, most of the previous convolutional neural network accelerator designs focused on solving the bottleneck of computational performance and bandwidth, ignoring the importance of convolutional neural network scarcity for accelerator design. In recent years, there are a few convolutional neural network accelerators that can take advantage of the scarcity, but they are usually difficult to consider in terms of computational flexibility, parallel efficiency and resource overhead. In view of the problem that the application of convolutional neural network (CNN) on the embedded side is limited by real-time, and there is a large degree of sparsity in CNN convolution calculation. This paper summarizes the methods of sparsification from the algorithm level and based on FPGA level. The different methods of sparsification and the research and analysis of different application layers are introduced. The advantages and development trend of sparsification are analyzed and summarized.

FPGA Hardware Acceleration Design for Deep Learning

A type of artificial neural network called a convolutional neural network (CNN) can learn characteristics from a huge amount of data and performs very well in the field of large-scale image processing. CNN simulates the behavior of a biological optic nerve. In recent years, with the development of deep neural network algorithms and hardware technology, the current "CPU+GPU" model servers cannot meet the neural network structure in various fields, so a large amount of deep CNN accelerators based on the FPGA platform have gradually emerged. FPGA is beginning to be used in the fields of image recognition and natural language processing because of its programmability, high performance, high stability, high security, and low power consumption. Though FPGA has proven to have better performance, there is still room for optimization at the design level. Yolov3, as a classical algorithm, still consumes a lot of time and computational resources in actual operations. To address this problem, this experiment partially optimizes the Yolov3 algorithm by introducing the CBAM attention mechanism in the Yolov3 model and pruning the embedded system with different proportions using the Network slimming method. Finally, it is verified on a TX2 embedded device developed by Nvidia using the COCO dataset. The experiment finds that the precision, mAP, and the number of parameters of the optimized Yolov3 algorithm under different optimization strategies. It is shown that the Yolov3 algorithm still has more optimization strategies that can reduce the time required for computation and the memory occupied more effectively without any degradation in accuracy.

A Convolutional Neural Network Accelerator Based on FPGA

This paper analyzes and studies the hardware programmable logic resources on small-scale FPGA chips, providing reasonable hardware resource support for subsequent neural network accelerator designs. A flexible 32-bit instruction set is designed for control by the Processing System (PS) on the Programmable Logic (PL) side, making motion state detection flexible and controllable. When designing the hardware side, this paper uses a resource-sharing strategy, and most of the calculation modules are designed using on-chip DSP resources to reduce the resource consumption of the calculation module. An innovative strategy of partially not caching the data between layers of the neural network is applied to reduce the demand for on-chip cache. To optimize on-chip storage space, this article partitions the limited BRAM space on the chip in a reasonable manner and improves the efficiency of on-chip data reading and writing through parallel processing, thereby improving the real-time performance of the neural network.

Compressing convolutional neural networks with cheap convolutions and online distillation

Visual impairment assistance systems play a vital role in improving the standard of living for visually impaired people (VIP). With the development of deep learning technologies and assistive devices, many assistive technologies for VIP have achieved remarkable success in environmental perception and navigation. In particular, convolutional neural network (CNN)-based models have surpassed the level of human recognition and achieved a strong generalization ability. However, the large memory and computation consumption in CNNs have been one of the main barriers to deploying them into resource-limited systems for visual impairment assistance applications. To this end, most cheap convolutions (e.g., group convolution, depth-wise convolution, and shift convolution) have recently been used for memory and computation reduction but with a specific architecture design. Furthermore, it results in a low discriminability of the compressed networks by directly replacing the standard convolution with these cheap ones. In this paper, we propose to use knowledge distillation to improve the performance of compact student networks with cheap convolutions. In our case, the teacher is a network with the standard convolution, while the student is a simple transformation of the teacher architecture without complicated redesigning. In particular, we introduce a novel online distillation method, which online constructs the teacher network without pre-training and conducts mutual learning between the teacher and student network, to improve the performance of the student model. Extensive experiments demonstrate that the proposed approach achieves superior performance to simultaneously reduce memory and computation overhead of cutting-edge CNNs on different datasets, including CIFAR-10/100 and ImageNet ILSVRC 2012, compared to the previous CNN compression and acceleration methods. The codes are publicly available at https://github.com/EthanZhangYC/OD-cheap-convolution.

A time domain 2D OaA-based convolutional neural networks accelerator

Convolutional neural networks (CNNs) are widely implemented in modern facial recognition systems for image recognition applications. Runtime speed is a critical parameter for real-time systems. Traditional FPGA-based accelerations require either large on-chip memory or high bandwidth and high memory access time that slow down the network. The proposed work uses an algorithm and its subsequent hardware design for a quick CNN computation using an overlap-and-add-based technique in the time domain. In the algorithm, the input images are broken into tiles that can be processed independently without computing overhead in the frequency domain. This also allows for efficient concurrency of the convolution process, resulting in higher throughput and lower power consumption. At the same time, we maintain low on-chip memory requirements necessary for faster and cheaper processor designs. We implemented CNN VGG-16 and AlexNet models with our design on Xilinx Virtex-7 and Zynq boards. The performance analysis of our design provides 48% better throughput than the state-of-the-art AlexNet and uses 68.85% lesser multipliers and other resources than the state-of-the-art VGG-16.

Design and Implementation of Convolutional Neural Network Accelerator Based on FPGA

Convolutional neural network, as a kind of feed-forward neural network, has been widely used in image recognition, speech processing and other fields in recent years. In this paper, an FPGA-based CNN gas pedal is designed to solve the problem of slow running and high-power consumption of CNN on resource-constrained hardware. The design invokes multi-stage pipeline parallel processing technology to accelerate convolutional operations; quantifies network parameters from 32-bit floating-point to 8-bit fixed-point while guaranteeing CNN accuracy, and uses data multiplexing to reduce resource consumption. Experimental results show that the design is 10 times faster than the intel i7-8700 and consumes only 1% of the power of the RTX 2060 at 50MHz.

DDAM: D ata D istribution- A ware M apping of CNNs on Processing-In-Memory Systems

Convolution neural networks (CNNs) are widely used algorithms in image processing, natural language processing and many other fields. The large amount of memory access of CNNs is one of the major concerns in CNN accelerator designs that influences the performance and energy-efficiency. With fast and low-cost memory access, Processing-In-Memory (PIM) system is a feasible solution to alleviate the memory concern of CNNs. However, the distributed manner of data storing in PIM systems is in conflict with the large amount of data reuse of CNN layers. Nodes of PIM systems may need to share their data with each other before processing a CNN layer, leading to extra communication overhead. In this article, we propose DDAM to map CNNs onto PIM systems with the communication overhead reduced. Firstly, A data transfer strategy is proposed to deal with the data sharing requirement among PIM nodes by formulating a Traveling-Salesman-Problem (TSP). To improve data locality, a dynamic programming algorithm is proposed to partition the CNN and allocate a number of nodes to each part. Finally, an integer linear programming (ILP)-based mapping algorithm is proposed to map the partitioned CNN onto the PIM system. Experimental results show that compared to the baselines, DDAM can get a higher throughput of 2.0× with the energy cost reduced by 37% on average.

FlexCNN: An End-to-end Framework for Composing CNN Accelerators on FPGA

With reduced data reuse and parallelism, recent convolutional neural networks (CNNs) create new challenges for FPGA acceleration. Systolic arrays (SAs) are efficient, scalable architectures for convolutional layers, but without proper optimizations, their efficiency drops dramatically for reasons: (1) the different dimensions within same-type layers, (2) the different convolution layers especially transposed and dilated convolutions, and (3) CNN’s complex dataflow graph. Furthermore, significant overheads arise when integrating FPGAs into machine learning frameworks. Therefore, we present a flexible, composable architecture called FlexCNN, which delivers high computation efficiency by employing dynamic tiling, layer fusion, and data layout optimizations. Additionally, we implement a novel versatile SA to process normal, transposed, and dilated convolutions efficiently. FlexCNN also uses a fully pipelined software-hardware integration that alleviates the software overheads. Moreover, with an automated compilation flow, FlexCNN takes a CNN in the ONNX 1 representation, performs a design space exploration, and generates an FPGA accelerator. The framework is tested using three complex CNNs: OpenPose, U-Net, and E-Net. The architecture optimizations achieve 2.3× performance improvement. Compared to a standard SA, the versatile SA achieves close-to-ideal speedups, with up to 5.98× and 13.42× for transposed and dilated convolutions, with a 6% average area overhead. The pipelined integration leads to a 5× speedup for OpenPose.

Towards Reconfigurable CNN Accelerator for FPGA Implementation

Convolutional Neural Networks (CNNs) have revolutionized many applications in recent years, especially in image classification, video processing, and pattern recognition. This success of CNNs has been a motivating factor for solving even more complex problems involving multiple data modalities. Traditionally, a single CNN accelerator has been optimized for just one task or has been used to perform correlated tasks. We leverage the CNNs capability to learn patterns and use one accelerator to perform multiple uncorrelated tasks from different modalities and achieve an average accuracy above 90%, which would otherwise require three accelerators. Two types of CNN architectures (i.e., fused and branched) are evaluated for three distinct tasks based on accuracy, quantization, pruning, hardware resource utilization, power, and latency. Capitalizing on this, we have further proposed a runtime reconfigurable CNN accelerator supporting fault-tolerant (FT), high-performance (HP), and de-stress (DS) modes.

Area-Efficient Parallel Multiplication Units for CNN Accelerators With Output Channel Parallelization

Many existing studies on accelerating convolutional neural networks (CNNs) use parallel data operation schemes to increase the throughput. This study proposes area-efficient parallel multiplication unit (PMU) designs for a CNN accelerator that uses parallelization on the output channels of a CNN layer, which parallel multiplies a common feature map pixel with multiple CNN kernel weights. First, tailored PMU designs are proposed for CNNs with specific low-precision 3-to-8-bit weights. Second, the proposed 5-to-8-bit PMU designs are extended with two-clock-cycle operations to develop PMUs for weight precision scalable to 10/12/14/16 bits. Compared to 16-path PMUs directly using carry-save-adder array multipliers, our PMU designs can achieve the area reductions of 28.19%−56.09% and 22.10%−30.71% for 3–8 bit and 10-/12-/14-/ 16-bit weights, respectively. Moreover, a resultant 16-path 16-bit weight PMU is verified through the system-on-chip (SoC) field-programmable gate array (FPGA) implementation to demonstrate the CNN inference.

Hybrid Multilevel STT/DSHE Memory for Efficient CNN Training

Convolutional neural network (CNN) is one of the most popular deep learning algorithms for artificial intelligence applications. However, due to its data-intensive workloads, the training of deep CNN is constrained by large memory requirements. Previously, spin transfer torque (STT)-based multilevel cells (MLCs) have been used to implement CNN accelerators to address this issue. However, due to multistep write and read operations, energy, latency, and reliability are critical constraints for larger and deeper training models. This article presents a hybrid STT and differential spin Hall effect (DSHE)-MLC-based data-aware memory design to achieve single-step write and read operations. Depending on the data type, the two-bit data are stored in either STT-MLC or DSHE-MLC memory blocks. The proposed scheme improves the training energy and latency in VGG-16, VGG-19, AlexNet, and LeNet CNN architectures by nearly 50% when compared with conventional STT-MLC-based architectures. Moreover, it also allows for the resolution of device-level write and read reliability issues.

Parallel Photonic Convolutional Processing on-Chip With Cross-Connect Architecture and Cyclic AWGs

Convolutional neural network (CNN) is one of the best neural network structures for solving classification problems. The convolutional processing of the network dominates processing time and computing power. Parallel computing for convolutional processing is essential to accelerate the computing speed of the neural network. In this paper, we introduce another domain of parallelism on top of the already demonstrated parallelisms suggested for photonic integrated processors with WDM approaches, to further accelerate the convolutional operation on chip. The operation of the novel parallelism is introduced with an updated cross-connect architecture, exploiting cyclic array waveguide grating. The photonic CNN system is demonstrated for the handwritten digit classification problem in simulation, with a speed of 2.56 Tera operation/s and end-to-end system energy efficiency of 3.75 pJ/operation, using 16 weighting elements and 10 Giga sample/s inputs. The proposed parallelism improves CNN acceleration by 4-16 times with respect to state-of-the-art integrated convolutional processors, depending on the available weighting elements per convolutional core.