Convolutional Neural Network Accelerator Research Articles

SoC-based real-time SVM classification with integrated training using HLS and PYNQ

Support Vector Machines (SVM) are widely used techniques in the field of classification problems because of their ability to effectively deal with datasets that have complex non-linear structures and a high dimensionality. The compute-intensive training algorithm associated with SVM makes it challenging to keep an up-to-date model that accurately reflects the characteristics of newly arriving data points in real-time systems. This paper proposes a novel training algorithm for incremental learning from large datasets, based on a variant of Sequential Minimal Optimization (SMO). High-Level Synthesis (HLS) was used for implementing the Field Programmable Gate Array (FPGA) based Intellectual Property (IP) Core, which includes the computationally intensive kernel computation portion of the training algorithm. In addition to the kernel computation, the inference phase of the SVM classifier is built into the IP core, and its use can be switched on the fly. The computational latency and memory bandwidth of an IP core are optimized using loop pipelining and DMA burst data transfer. With the help of hardware/software co-design, the IP core is integrated into the design of a flexible and re-usable System on Chip (SoC) called PYNQ Overlay. The experiments show that the overlay outperforms the embedded processor, multiple hardware SVM classifiers, and hardware accelerated Convolutional Neural Networks (CNN) in terms of real-time efficiency. The Overlay makes much less use of the resources available on the chip in comparison to the majority of the CNN accelerators. The overlay achieves an average classification accuracy that is only 1% lower than that of an ARM Cortex-A9 processor, according to experimental results on six datasets. Furthermore, it can increase training speed by an average of 31.82x and inference speed by an average of 31.74x. In addition, the proposed Overlay design achieves a 2.3x improvement in average training speed, as measured in Mega bits per second, compared to existing SVM training implementations, along with incremental learning and multi-class classification support.

X-NVDLA: Runtime Accuracy Configurable NVDLA Based on Applying Voltage Overscaling to Computing and Memory Units

This paper investigates a runtime accuracy reconfigurable implementation of an energy efficient deep learning accelerator. It is based on voltage overscaling (VOS) technique which provides dynamic adjustment of approximation level as well as improving lifetime/reliability of the accelerator. The technique is applied to both computing and memory units where based on the minimum required accuracy, the applied voltage is adjusted during the runtime. The implementation of the network is performed using NVDLA which is an open-source CNN (Convolutional Neural Network) accelerator. The approximation is applied to both the accelerator MAC array employed for the required network computations and to the accelerator SRAM memory utilized for storing the inputs (images), weights, and activation data. To control the accuracy degradation of the approximate accelerator (called X-NVDLA), the reduced voltage is applied only to LSB bits of the MAC array and SRAM unit. To assess the efficacy of the proposed energy efficient accelerator, the energy-accuracy characteristics of X-NVDLA when running LeNet-5 and ResNet-50 networks with 8-bit (integer) precision are investigated. In addition, the characteristic of bias temperature instability (BTI), as one of the lifetime deteriorating phenomena is determined. The study includes energy improvement versus accuracy degradation as a function of overscaled voltages and number of approximate the least significant bits using a 15nm FinFET technology.

CODEBench: A Neural Architecture and Hardware Accelerator Co-Design Framework

Recently, automated co-design of machine learning (ML) models and accelerator architectures has attracted significant attention from both the industry and academia. However, most co-design frameworks either explore a limited search space or employ suboptimal exploration techniques for simultaneous design decision investigations of the ML model and the accelerator. Furthermore, training the ML model and simulating the accelerator performance is computationally expensive. To address these limitations, this work proposes a novel neural architecture and hardware accelerator co-design framework, called CODEBench. It comprises two new benchmarking sub-frameworks, CNNBench and AccelBench, which explore expanded design spaces of convolutional neural networks (CNNs) and CNN accelerators. CNNBench leverages an advanced search technique, Bayesian Optimization using Second-order Gradients and Heteroscedastic Surrogate Model for Neural Architecture Search, to efficiently train a neural heteroscedastic surrogate model to converge to an optimal CNN architecture by employing second-order gradients. AccelBench performs cycle-accurate simulations for diverse accelerator architectures in a vast design space. With the proposed co-design method, called Bayesian Optimization using Second-order Gradients and Heteroscedastic Surrogate Model for Co-Design of CNNs and Accelerators, our best CNN–accelerator pair achieves 1.4% higher accuracy on the CIFAR-10 dataset compared to the state-of-the-art pair while enabling 59.1% lower latency and 60.8% lower energy consumption. On the ImageNet dataset, it achieves 3.7% higher Top1 accuracy at 43.8% lower latency and 11.2% lower energy consumption. CODEBench outperforms the state-of-the-art framework, i.e., Auto-NBA, by achieving 1.5% higher accuracy and 34.7× higher throughput while enabling 11.0× lower energy-delay product and 4.0× lower chip area on CIFAR-10.

Design of a Low-Power Super-Resolution Architecture for Virtual Reality Wearable Devices

Head-mounted displays (HMDs) have made a virtual reality (VR) accessible to a widespread consumer market, introducing a revolution in many applications. Among the limitations of current HMD technology, the need for generating high-resolution images and streaming them at adequate frame rates is one of the most critical. Super-resolution (SR) convolutional neural networks (CNNs) can be exploited to alleviate timing and bandwidth bottlenecks of video streaming by reconstructing high-resolution images locally (i.e., near the display). However, such techniques involve a significant amount of computations that makes their deployment within area-/power-constrained wearable devices often unfeasible. This research work originated from the consideration that the human eye can capture details with high acuity only within a certain region, called the fovea. Therefore, we designed a custom hardware architecture able to reconstruct high-resolution images by treating foveal region (FR) and peripheral region (PR) through accurate and inaccurate operations, respectively. Hardware experiments demonstrate the effectiveness of our proposal: a customized fast SR CNN (FSRCNN) accelerator realized as described here and implemented on a 28-nm process technology is able to process up to 214 ultrahigh definition frames/s, while consuming just 0.51 pJ/pixel without compromising the perceptual visual quality, thus achieving a 55% energy reduction and a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times 14$ </tex-math></inline-formula> times higher throughput rate, with respect to state-of-the-art competitors.

Sparse-Aware Deep Learning Accelerator

In view of the difficulty of hardware implementation of convolutional neural network computing, most of the previous convolutional neural network accelerator designs focused on solving the bottleneck of computational performance and bandwidth, ignoring the importance of convolutional neural network scarcity for accelerator design. In recent years, there are a few convolutional neural network accelerators that can take advantage of the scarcity, but they are usually difficult to consider in terms of computational flexibility, parallel efficiency and resource overhead. In view of the problem that the application of convolutional neural network (CNN) on the embedded side is limited by real-time, and there is a large degree of sparsity in CNN convolution calculation. This paper summarizes the methods of sparsification from the algorithm level and based on FPGA level. The different methods of sparsification and the research and analysis of different application layers are introduced. The advantages and development trend of sparsification are analyzed and summarized.

FPGA Hardware Acceleration Design for Deep Learning

A type of artificial neural network called a convolutional neural network (CNN) can learn characteristics from a huge amount of data and performs very well in the field of large-scale image processing. CNN simulates the behavior of a biological optic nerve. In recent years, with the development of deep neural network algorithms and hardware technology, the current "CPU+GPU" model servers cannot meet the neural network structure in various fields, so a large amount of deep CNN accelerators based on the FPGA platform have gradually emerged. FPGA is beginning to be used in the fields of image recognition and natural language processing because of its programmability, high performance, high stability, high security, and low power consumption. Though FPGA has proven to have better performance, there is still room for optimization at the design level. Yolov3, as a classical algorithm, still consumes a lot of time and computational resources in actual operations. To address this problem, this experiment partially optimizes the Yolov3 algorithm by introducing the CBAM attention mechanism in the Yolov3 model and pruning the embedded system with different proportions using the Network slimming method. Finally, it is verified on a TX2 embedded device developed by Nvidia using the COCO dataset. The experiment finds that the precision, mAP, and the number of parameters of the optimized Yolov3 algorithm under different optimization strategies. It is shown that the Yolov3 algorithm still has more optimization strategies that can reduce the time required for computation and the memory occupied more effectively without any degradation in accuracy.

A Convolutional Neural Network Accelerator Based on FPGA

This paper analyzes and studies the hardware programmable logic resources on small-scale FPGA chips, providing reasonable hardware resource support for subsequent neural network accelerator designs. A flexible 32-bit instruction set is designed for control by the Processing System (PS) on the Programmable Logic (PL) side, making motion state detection flexible and controllable. When designing the hardware side, this paper uses a resource-sharing strategy, and most of the calculation modules are designed using on-chip DSP resources to reduce the resource consumption of the calculation module. An innovative strategy of partially not caching the data between layers of the neural network is applied to reduce the demand for on-chip cache. To optimize on-chip storage space, this article partitions the limited BRAM space on the chip in a reasonable manner and improves the efficiency of on-chip data reading and writing through parallel processing, thereby improving the real-time performance of the neural network.

Compressing convolutional neural networks with cheap convolutions and online distillation

Visual impairment assistance systems play a vital role in improving the standard of living for visually impaired people (VIP). With the development of deep learning technologies and assistive devices, many assistive technologies for VIP have achieved remarkable success in environmental perception and navigation. In particular, convolutional neural network (CNN)-based models have surpassed the level of human recognition and achieved a strong generalization ability. However, the large memory and computation consumption in CNNs have been one of the main barriers to deploying them into resource-limited systems for visual impairment assistance applications. To this end, most cheap convolutions (e.g., group convolution, depth-wise convolution, and shift convolution) have recently been used for memory and computation reduction but with a specific architecture design. Furthermore, it results in a low discriminability of the compressed networks by directly replacing the standard convolution with these cheap ones. In this paper, we propose to use knowledge distillation to improve the performance of compact student networks with cheap convolutions. In our case, the teacher is a network with the standard convolution, while the student is a simple transformation of the teacher architecture without complicated redesigning. In particular, we introduce a novel online distillation method, which online constructs the teacher network without pre-training and conducts mutual learning between the teacher and student network, to improve the performance of the student model. Extensive experiments demonstrate that the proposed approach achieves superior performance to simultaneously reduce memory and computation overhead of cutting-edge CNNs on different datasets, including CIFAR-10/100 and ImageNet ILSVRC 2012, compared to the previous CNN compression and acceleration methods. The codes are publicly available at https://github.com/EthanZhangYC/OD-cheap-convolution.

A time domain 2D OaA-based convolutional neural networks accelerator

Convolutional neural networks (CNNs) are widely implemented in modern facial recognition systems for image recognition applications. Runtime speed is a critical parameter for real-time systems. Traditional FPGA-based accelerations require either large on-chip memory or high bandwidth and high memory access time that slow down the network. The proposed work uses an algorithm and its subsequent hardware design for a quick CNN computation using an overlap-and-add-based technique in the time domain. In the algorithm, the input images are broken into tiles that can be processed independently without computing overhead in the frequency domain. This also allows for efficient concurrency of the convolution process, resulting in higher throughput and lower power consumption. At the same time, we maintain low on-chip memory requirements necessary for faster and cheaper processor designs. We implemented CNN VGG-16 and AlexNet models with our design on Xilinx Virtex-7 and Zynq boards. The performance analysis of our design provides 48% better throughput than the state-of-the-art AlexNet and uses 68.85% lesser multipliers and other resources than the state-of-the-art VGG-16.

Design and Implementation of Convolutional Neural Network Accelerator Based on FPGA

Convolutional neural network, as a kind of feed-forward neural network, has been widely used in image recognition, speech processing and other fields in recent years. In this paper, an FPGA-based CNN gas pedal is designed to solve the problem of slow running and high-power consumption of CNN on resource-constrained hardware. The design invokes multi-stage pipeline parallel processing technology to accelerate convolutional operations; quantifies network parameters from 32-bit floating-point to 8-bit fixed-point while guaranteeing CNN accuracy, and uses data multiplexing to reduce resource consumption. Experimental results show that the design is 10 times faster than the intel i7-8700 and consumes only 1% of the power of the RTX 2060 at 50MHz.

DDAM: D ata D istribution- A ware M apping of CNNs on Processing-In-Memory Systems

Convolution neural networks (CNNs) are widely used algorithms in image processing, natural language processing and many other fields. The large amount of memory access of CNNs is one of the major concerns in CNN accelerator designs that influences the performance and energy-efficiency. With fast and low-cost memory access, Processing-In-Memory (PIM) system is a feasible solution to alleviate the memory concern of CNNs. However, the distributed manner of data storing in PIM systems is in conflict with the large amount of data reuse of CNN layers. Nodes of PIM systems may need to share their data with each other before processing a CNN layer, leading to extra communication overhead. In this article, we propose DDAM to map CNNs onto PIM systems with the communication overhead reduced. Firstly, A data transfer strategy is proposed to deal with the data sharing requirement among PIM nodes by formulating a Traveling-Salesman-Problem (TSP). To improve data locality, a dynamic programming algorithm is proposed to partition the CNN and allocate a number of nodes to each part. Finally, an integer linear programming (ILP)-based mapping algorithm is proposed to map the partitioned CNN onto the PIM system. Experimental results show that compared to the baselines, DDAM can get a higher throughput of 2.0× with the energy cost reduced by 37% on average.

FlexCNN: An End-to-end Framework for Composing CNN Accelerators on FPGA

With reduced data reuse and parallelism, recent convolutional neural networks (CNNs) create new challenges for FPGA acceleration. Systolic arrays (SAs) are efficient, scalable architectures for convolutional layers, but without proper optimizations, their efficiency drops dramatically for reasons: (1) the different dimensions within same-type layers, (2) the different convolution layers especially transposed and dilated convolutions, and (3) CNN’s complex dataflow graph. Furthermore, significant overheads arise when integrating FPGAs into machine learning frameworks. Therefore, we present a flexible, composable architecture called FlexCNN, which delivers high computation efficiency by employing dynamic tiling, layer fusion, and data layout optimizations. Additionally, we implement a novel versatile SA to process normal, transposed, and dilated convolutions efficiently. FlexCNN also uses a fully pipelined software-hardware integration that alleviates the software overheads. Moreover, with an automated compilation flow, FlexCNN takes a CNN in the ONNX 1 representation, performs a design space exploration, and generates an FPGA accelerator. The framework is tested using three complex CNNs: OpenPose, U-Net, and E-Net. The architecture optimizations achieve 2.3× performance improvement. Compared to a standard SA, the versatile SA achieves close-to-ideal speedups, with up to 5.98× and 13.42× for transposed and dilated convolutions, with a 6% average area overhead. The pipelined integration leads to a 5× speedup for OpenPose.

Towards Reconfigurable CNN Accelerator for FPGA Implementation

Convolutional Neural Networks (CNNs) have revolutionized many applications in recent years, especially in image classification, video processing, and pattern recognition. This success of CNNs has been a motivating factor for solving even more complex problems involving multiple data modalities. Traditionally, a single CNN accelerator has been optimized for just one task or has been used to perform correlated tasks. We leverage the CNNs capability to learn patterns and use one accelerator to perform multiple uncorrelated tasks from different modalities and achieve an average accuracy above 90%, which would otherwise require three accelerators. Two types of CNN architectures (i.e., fused and branched) are evaluated for three distinct tasks based on accuracy, quantization, pruning, hardware resource utilization, power, and latency. Capitalizing on this, we have further proposed a runtime reconfigurable CNN accelerator supporting fault-tolerant (FT), high-performance (HP), and de-stress (DS) modes.

Area-Efficient Parallel Multiplication Units for CNN Accelerators With Output Channel Parallelization

Many existing studies on accelerating convolutional neural networks (CNNs) use parallel data operation schemes to increase the throughput. This study proposes area-efficient parallel multiplication unit (PMU) designs for a CNN accelerator that uses parallelization on the output channels of a CNN layer, which parallel multiplies a common feature map pixel with multiple CNN kernel weights. First, tailored PMU designs are proposed for CNNs with specific low-precision 3-to-8-bit weights. Second, the proposed 5-to-8-bit PMU designs are extended with two-clock-cycle operations to develop PMUs for weight precision scalable to 10/12/14/16 bits. Compared to 16-path PMUs directly using carry-save-adder array multipliers, our PMU designs can achieve the area reductions of 28.19%−56.09% and 22.10%−30.71% for 3–8 bit and 10-/12-/14-/ 16-bit weights, respectively. Moreover, a resultant 16-path 16-bit weight PMU is verified through the system-on-chip (SoC) field-programmable gate array (FPGA) implementation to demonstrate the CNN inference.

Hybrid Multilevel STT/DSHE Memory for Efficient CNN Training

Convolutional neural network (CNN) is one of the most popular deep learning algorithms for artificial intelligence applications. However, due to its data-intensive workloads, the training of deep CNN is constrained by large memory requirements. Previously, spin transfer torque (STT)-based multilevel cells (MLCs) have been used to implement CNN accelerators to address this issue. However, due to multistep write and read operations, energy, latency, and reliability are critical constraints for larger and deeper training models. This article presents a hybrid STT and differential spin Hall effect (DSHE)-MLC-based data-aware memory design to achieve single-step write and read operations. Depending on the data type, the two-bit data are stored in either STT-MLC or DSHE-MLC memory blocks. The proposed scheme improves the training energy and latency in VGG-16, VGG-19, AlexNet, and LeNet CNN architectures by nearly 50% when compared with conventional STT-MLC-based architectures. Moreover, it also allows for the resolution of device-level write and read reliability issues.

Parallel Photonic Convolutional Processing on-Chip With Cross-Connect Architecture and Cyclic AWGs

Convolutional neural network (CNN) is one of the best neural network structures for solving classification problems. The convolutional processing of the network dominates processing time and computing power. Parallel computing for convolutional processing is essential to accelerate the computing speed of the neural network. In this paper, we introduce another domain of parallelism on top of the already demonstrated parallelisms suggested for photonic integrated processors with WDM approaches, to further accelerate the convolutional operation on chip. The operation of the novel parallelism is introduced with an updated cross-connect architecture, exploiting cyclic array waveguide grating. The photonic CNN system is demonstrated for the handwritten digit classification problem in simulation, with a speed of 2.56 Tera operation/s and end-to-end system energy efficiency of 3.75 pJ/operation, using 16 weighting elements and 10 Giga sample/s inputs. The proposed parallelism improves CNN acceleration by 4-16 times with respect to state-of-the-art integrated convolutional processors, depending on the available weighting elements per convolutional core.

Systematic Reliability Evaluation of FPGA Implemented CNN Accelerators

Convolutional neural networks (CNN) have become essential for many scientific and industrial applications, such as image classification and pattern detection. Among the devices that can implement neural networks, SRAM based FPGAs are a popular option due to their excellent parallel computing capability and good flexibility. However, SRAM-FPGAs are susceptible to radiation effects, which limits its application on safety critical applications. In this paper, the reliability of an accelerator based on the advanced Instruction-Set Architecture is evaluated based on hardware fault injection experiments. Each main module of the accelerator is evaluated separately, and the impact of parallelism and model features on the accelerator reliability is also examined. The experimental results reveal some important conclusions in terms of general hardware reliability and also of the particular model reliability. First, over 99% of SEUs on the computation modules will cause accuracy loss, and the reliability improves for higher parallelism. Second, a large portion of SEUs on the data mover and the instruction scheduler will cause system corruptions due to abnormal interactions with the ARM or other modules. Third, nonlinear activation and pooling layers are effective in reducing the effect of SEUs on computation modules, so models that use these layers tend to be more robust. The results provide a deep understanding of the impact of errors on CNNs implemented on ISA based FPGA accelerators (e.g., the Xilinx DPU).

Efficient Layer-Wise N:M Sparse CNN Accelerator with Flexible SPEC: Sparse Processing Element Clusters

Recently, the layer-wise N:M fine-grained sparse neural network algorithm (i.e., every M-weights contains N non-zero values) has attracted tremendous attention, as it can effectively reduce the computational complexity with negligible accuracy loss. However, the speed-up potential of this algorithm will not be fully exploited if the right hardware support is lacking. In this work, we design an efficient accelerator for the N:M sparse convolutional neural networks (CNNs) with layer-wise sparse patterns. First, we analyze the performances of different processing element (PE) structures and extensions to construct the flexible PE architecture. Second, the variable sparse convolutional dimensions and sparse ratios are involved in the hardware design. With a sparse PE cluster (SPEC) design, the hardware can efficiently accelerate CNNs with the layer-wise N:M pattern. Finally, we employ the proposed SPEC into the CNN accelerator with flexible network-on-chip and specially designed dataflow. We implement hardware accelerators on Xilinx ZCU102 FPGA and Xilinx VCU118 FPGA and evaluate them with classical CNNs such as Alexnet, VGG-16, and ResNet-50. Compared with existing accelerators designed for structured and unstructured pruned networks, our design achieves the best performance in terms of power efficiency.

Design of Low-Complexity Convolutional Neural Network Accelerator for Finger Vein Identification System.

In the biometric field, vein identification is a vital process that is constrained by the invisibility of veins as well as other unique features. Moreover, users generally do not wish to have their personal information uploaded to the cloud, so edge computing has become popular for the sake of protecting user privacy. In this paper, we propose a low-complexity and lightweight convolutional neural network (CNN) and we design intellectual property (IP) for shortening the inference time in finger vein recognition. This neural network system can operate independently in client mode. After fetching the user's finger vein image via a near-infrared (NIR) camera mounted on an embedded system, vein features can be efficiently extracted by vein curving algorithms and user identification can be completed quickly. Better image quality and higher recognition accuracy can be obtained by combining several preprocessing techniques and the modified CNN. Experimental data were collected by the finger vein image capture equipment developed in our laboratory based on the specifications of similar products currently on the market. Extensive experiments demonstrated the practicality and robustness of the proposed finger vein identification system.

A Low-Power Hardware Architecture for Real-Time CNN Computing.

Convolutional neural network (CNN) is widely deployed on edge devices, performing tasks such as objective detection, image recognition and acoustic recognition. However, the limited resources and strict power constraints of edge devices pose a great challenge to applying the computationally intensive CNN models. In addition, for the edge applications with real-time requirements, such as real-time computing (RTC) systems, the computations need to be completed considering the required timing constraint, so it is more difficult to trade off between computational latency and power consumption. In this paper, we propose a low-power CNN accelerator for edge inference of RTC systems, where the computations are operated in a column-wise manner, to realize an immediate computation for the currently available input data. We observe that most computations of some CNN kernels in deep layers can be completed in multiple cycles, while not affecting the overall computational latency. Thus, we present a multi-cycle scheme to conduct the column-wise convolutional operations to reduce the hardware resource and power consumption. We present hardware architecture for the multi-cycle scheme as a domain-specific CNN architecture, which is then implemented in a 65 nm technology. We prove our proposed approach realizes up to 8.45%, 49.41% and 50.64% power reductions for LeNet, AlexNet and VGG16, respectively. The experimental results show that our approach tends to cause a larger power reduction for the CNN models with greater depth, larger kernels and more channels.