Neural Accelerator Research Articles

Neural acceleration of incomplete factorization preconditioning

Neural acceleration of incomplete factorization preconditioning

Design and Implementation of a Universal Shift Convolutional Neural Network Accelerator

Currently, many applications implement convolutional neural networks on CPUs or GPUs while the performances are limited by the computational complexity of these networks. Compared with the implementation on CPUs or GPUs, deploying convolutional neural accelerators on FPGAs can achieve superior performance. On the other side, the multiplication operations of convolutional neural networks have been a constraint for FPGAs to achieve the better performance. In this paper, we proposed a shift convolutional neural network accelerator, which converts the multiplication operations into shift operations. Based on the shift operation, our accelerator can break the computational bottleneck of FPGAs. On Virtex UltraScale+ VU9P, our accelerator can save DSP resources and reduce memory consumption while achieving a performance of 1.18 TOPS(Tera Operations Per Second), which is an essential improvement over the other convolutional neural accelerators.

Adaptive neural acceleration unit based on heterogeneous multicore hardware architecture FPGA and software-defined hardware

ABSTRACT This study is anchored in a heterogeneous multicore hardware architecture, specifically Field Programmable Gate Arrays (FPGAs), and software-defined hardware. It employs dynamic planning for the reconfiguration of a software-defined System-on-Chip (SOC) to expedite neural network processes. The reconfigured software-defined SOC serves as an efficient data processor, facilitating the hybrid development and verification of domain-specific System-on-Chip architectures, adhering to the Advanced Microcontroller Bus Architecture (AMBA) standard. The proposed block data trimming methodology significantly enhances overall hybrid computing efficiency by mitigating throughput limitations inherent in Systolic array hardware. The designed Systolic array Matrix Multiply Unit (MMU) is evaluated for a maximum MMU size of 32 × 32 and 1,024 Multiply Accumulator (MAC) units. Hybrid dynamic circuits are devised for the synthesis of int8, int16, int32, and int64 data types and associated logic circuits to validate the performance of parallel computing. It is anticipated that this proposed system can find utility in the development and deployment of unmanned devices, as well as in the integration of deep learning and multi-sensory fusion involving acoustic, tactile, and force sensory components.

Impact of TID on the Analog Conductance and Training Accuracy of CBRAM-Based Neural Accelerator

Impact of TID on the Analog Conductance and Training Accuracy of CBRAM-Based Neural Accelerator

Neural Acceleration of Graph Based Utility Functions for Sparse Matrices

Many graph-based algorithms in high performance computing (HPC) use approximate solutions due to having algorithms that are computationally expensive or serial in nature. Neural acceleration, i.e., the process of speeding up approximation computation elements via artificial neural networks, is relatively new and has not focused on HPC graph-based algorithms. In this paper, we propose a starting point for applying models for neural acceleration to graph-based HPC algorithms utilizing an understanding of the connectivity computational pattern, recursive neural networks, and graph neural networks. We demonstrate these techniques on the problem related to the utility functions of sparse matrix ordering and fill-in (i.e., zero elements becoming nonzero during factorization) calculations. The problem of sparse matrix ordering is commonly used for issues related to load balancing, improving memory reuse, or reducing computational and memory costs in direct sparse linear solver methods. These utility functions are ideal for demonstration as they comprise a number of different graph-based subproblems, and thus demonstrate the usefulness of our method over a wide range. We show that we can accurately approximate the best ordering and the nonzero count of the sparse factorization matrix while speeding up the calculation by as much as 30.3× over the traditional serial method.

A Study on the Design Procedure of Re-Configurable Convolutional Neural Network Engine for FPGA-Based Applications

Convolutional neural networks (CNNs) have become a primary approach in the field of artificial intelligence (AI), with wide range of applications. The two computational phases for every neural network are; the training phase and the testing phase. Usually, testing is performed on high-processing hardware engines, however, the training part is still a challenge for low-power devices. There are several neural accelerators; such as graphics processing units and field-programmable-gate-arrays (FPGAs). From the design perspective, an efficient hardware engine at the register-transfer level and efficient CNN modeling at the TensorFlow level are mandatory for any type of application. Hence, we propose a comprehensive, and step-by-step design procedure for a re-configurable CNN engine. We used TensorFlow and Keras libraries for modeling in Python, whereas the register-transfer-level part was performed using Verilog. The proposed idea was synthesized, placed, and routed for 180 nm complementary metal-oxide semiconductor technology using synopsis design compiler tools. The proposed design layout occupies an area of 3.16 × 3.16 mm2. A competitive accuracy of approximately 96% was achieved for the Modified National Institute of Standards and Technology (MNIST) and Canadian Institute for Advanced Research (CIFAR-10) datasets.

Dynamic Rate Neural Acceleration Using Multiprocessing Mode Support

Multiobject detection has become an integral component in various neural applications, such as autonomous driving and augmented reality. The system should be able to recognize and process multiple objects simultaneously. Moreover, the performance requirements for this system can be dynamically changed depending on the number of regions of interest (ROIs) in each frame. Consequently, the processing unit (PU) of the neural acceleration system should provide various inference rates. Therefore, we present a field-programmable gate array (FPGA)-based dynamic rate neural acceleration system called MultiLockOn to dynamically change the inference performance according to the number of ROIs per frame. It supports multiprocessing modes with different speeds through the introduction of novel multi-mode processing engines (PEs) comprising minimum reconfigurable interconnections across inference modes to minimize hardware overhead. The MultiLockOn system can provide an improvement of up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$4\times $ </tex-math></inline-formula> in the inference performance compared to that of DNNWeaver and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$5.7\times $ </tex-math></inline-formula> compared to that of the ARM Cortex-A53 with minimum accuracy loss by supporting the multiprocessing modes.

Characterizing Deep Neural Networks on Edge Computing Systems for Object Classification in 3D Point Clouds

The current trend of shifting computing from the cloud to the edge of the Internet of Things is influencing deep learning applications. Moving intelligence closer to the point of need entails advantages in terms of performance, power consumption, security, and privacy. The problem arises with data sources that generate a massive amount of information, making data processing challenging for edge devices. This is the case of point clouds generated by LIDAR sensors. Implementations at the edge become even more challenging when heavy processing algorithms such as deep neural networks are selected. However, deep neural networks are the state-of-the-art solution to carry out object classification tasks as they provide the best results in terms of accuracy when working with high data volumes. This work demonstrates that the processing of point cloud-based sensors using deep neural networks at the edge is becoming feasible with the emergence of new devices with high computing capacity combined with reduced power consumption. In this regard, a characterization of first-in-class deep learning classification algorithms working with point cloud data as inputs and running over different state-of-the-art edge processing architectures is provided. A broad range of devices, including CPUs, GPU-based, SoC FPGA-based, and deep learning neural accelerators, have been evaluated in terms of inference time, classification accuracy, and power consumption. As a result, it demonstrates that neural accelerators with integrated host CPUs represent the best trade-off between power consumption and performance, making them a perfect solution for IoT applications at the edge level.

RNA: A Flexible and Efficient Accelerator Based on Dynamically Reconfigurable Computing for Multiple Convolutional Neural Networks

The increasingly complicated and versatile convolutional neural networks (CNNs) models bring challenges to hardware acceleration in terms of performance, energy efficiency and flexibility. This paper proposes a reconfigurable neural accelerator (RNA) for flexible and efficient CNN acceleration. To provide hardware flexibility, RNA employs dynamically reconfigurable computing framework to rapidly configure data path between processing elements (PE) at run-time, as well as an interlaced data access mechanism for multi-bank RAM. To achieve high energy efficiency, three optimization mechanisms, including image row broadcasting dataflow (IRBD), tile-by-tile computing (TTC), and zero detection technology (ZDT), are dedicatedly designed for RNA to exploit data reuse and decrease memory bandwidth requirement, which is the key to improving performance and saving power consumption. To save hardware overhead, an online dynamic adaptive data truncation (DADT) mechanism is designed to compensate accuracy loss of multiplication results so that the computational precision in RNA can be reduced from 16-bit to 8-bit, which contributes to reducing the area of data path. The RNA architecture is implemented on Xilinx XC7Z100 FPGA and works at 250[Formula: see text]MHz. Experimental results show that the performance of running LeNet, AlexNet and VGG are 500 GOPS, 598 GOPS and 660 GOPS, respectively. Compared to previous FPGA-based designs, RNA achieves [Formula: see text] performance speedup and [Formula: see text] improvements on energy efficiency.

Light in AI: Toward Efficient Neurocomputing With Optical Neural Networks—A Tutorial

In the post Moore&#x2019;s era, conventional electronic digital computing platforms have encountered escalating challenges to support massively parallel and energy-hungry artificial intelligence (AI) workloads. Intelligent applications in data centers, edge devices, and autonomous vehicles have restricted requirements in throughput, power, and latency, which raises a high demand for a revolutionary neurocomputing solution. Optical neural network (ONN) is a promising hardware platform that could represent a paradigm shift in efficient neurocomputing with its ultra-fast speed, high parallelism, and low energy consumption. In recent years, efforts have been made to facilitate the ONN design stack and push forward the practical application of optical neural accelerators. In this tutorial, we give an overview of state-of-the-art cross-layer co-design methodologies for scalable, robust, and self-learnable ONN designs across the circuit, architecture, and algorithm levels. Besides, we analyze challenges and highlight emerging directions targeting next-generation optics for AI.

An Emotion and Attention Recognition System to Classify the Level of Engagement to a Video Conversation by Participants in Real Time Using Machine Learning Models and Utilizing a Neural Accelerator Chip

It is not an easy task for organizers to observe the engagement level of a video meeting audience. This research was conducted to build an intelligent system to enhance the experience of video conversations such as virtual meetings and online classrooms using convolutional neural network (CNN)- and support vector machine (SVM)-based machine learning models to classify the emotional states and the attention level of the participants to a video conversation. This application visualizes their attention and emotion analytics in a meaningful manner. This proposed system provides an artificial intelligence (AI)-powered analytics system with optimized machine learning models to monitor the audience and prepare insightful reports on the basis of participants’ facial features throughout the video conversation. One of the main objectives of this research is to utilize the neural accelerator chip to enhance emotion and attention detection tasks. A custom CNN developed by Gyrfalcon Technology Inc (GTI) named GnetDet was used in this system to run the trained model on their GTI Lightspeeur 2803 neural accelerator chip.

Low-Power Detection and Classification for In-Sensor Predictive Maintenance Based on Vibration Monitoring

In this work, a new custom design of an anomaly detection and classification system is proposed. It is composed of a convolutional Auto-Encoder (AE) hardware design to perform anomaly detection which cooperates with a mixed HW/SW Convolutional Neural Network (CNN) to perform the classification of detected anomalies. The AE features a partial binarization, so that the weights are binarized while the activations, associated to some selected layers, are non-binarized. This has been necessary to meet the severe area and energy constraints that allow it to be integrated on the same die as the MEMS sensors for which it serves as a neural accelerator. The CNN shares the feature extraction module with the AE, whereas a SW classifier is triggered by the AE when a fault is detected, working asynchronously to it. The AE has been mapped on a Xilinx Artix-7 FPGA, featuring an Output Data Rate (ODR) of 365 kHz and achieving a power dissipation of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$333 \boldsymbol {\mu } $ </tex-math></inline-formula> W/MHz. Logic synthesis has targeted TSMC CMOS 65 nm, 90 nm, and 130 nm standard cells. Best results achieved highlight a power consumption of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$138 \boldsymbol {\mu } $ </tex-math></inline-formula> W/MHz with an area occupation of 0.49 mm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> when real-time operations are set. These results enable the integration of the complete neural accelerator in the CMOS circuitry that typically sits with the inertial MEMS on the same silicon die. Comparisons with the related works suggest that the proposed system is capable of state-of-the-art performances and accuracy.

SCAN: Streamlined Composite Activation Function Unit for Deep Neural Accelerators

SCAN: Streamlined Composite Activation Function Unit for Deep Neural Accelerators

Toward Robust Facial Authentication for Low-Power Edge-AI Consumer Devices

Robust authentication for low-power consumer devices without a keyboard remains a challenge. The recent availability of low-power neural accelerator hardware, combined with improvements in neural facial recognition algorithms provides enabling technology for low-power, on-device facial authentication. The present research work explores a number of approaches to test the robustness of a state-of-the-art facial recognition (FR) technique, Arcface for such end-to-end applications. As extreme lighting conditions and facial pose are the two more challenging scenarios for FR we focus on these. Due to the general lack of large-scale multiple-identity datasets, GAN-based re-lighting and pose techniques are used to explore the effects on FR performance. These results are further validated on the best available multi-identity datasets - MultiPIE and BIWI. The results show that FR is quite robust to pose variations up to 45-55 degrees, but the outcomes are not definitive for the tested lighting scenarios. For lighting, the tested GAN-based relighting augmentations show significant effects on FR robustness. However, the lighting scenarios from MultiPIE dataset - the best available public dataset - show some conflicting results. It is unclear if this is due to an incorrectly learned GAN relighting transformation or, alternatively, to mixed ambient/directional lighting scenes in the dataset. However, it is shown that the GAN-induced FR errors for extreme lighting conditions can be corrected by fine-tuning the FR network layers. The conclusions support the feasibility of implementing a robust authentication method for low-power consumer devices.

A Drift-Resilient Hardware Implementation of Neural Accelerators Based on Phase Change Memory Devices

Memory devices, such as the phase change memory (PCM), have recently shown significant breakthroughs in terms of compactness, 3-D stacking capability, and speed up for deep learning neural accelerators. However, PCM is affected by the conductance drift, which prevents a precise definition of the synaptic weights in artificial neural networks. Here, we propose an efficient system-level methodology to develop drift-resilient multilayer perceptron (MLP) networks. The procedure guarantees high testing accuracy under conductance drift of the devices and enables the use of only positive weights. We validate the methodology using MNIST, rand-MNIST, and Fashion-MNIST datasets, thus offering a roadmap for the implementation of integrated nonvolatile memory-based neural networks. We finally analyze the proposed architecture in terms of throughput and energy efficiency. This work highlights the relevance of robust PCM-based design of neural networks for improving the computational capability and optimizing energetic efficiency.

Automated Pest Detection With DNN on the Edge for Precision Agriculture

Artificial intelligence has smoothly penetrated several economic activities, especially monitoring and control applications, including the agriculture sector. However, research efforts toward low-power sensing devices with fully functional machine learning (ML) on-board are still fragmented and limited in smart farming. Biotic stress is one of the primary causes of crop yield reduction. With the development of deep learning in computer vision technology, autonomous detection of pest infestation through images has become an important research direction for timely crop disease diagnosis. This paper presents an embedded system enhanced with ML functionalities, ensuring continuous detection of pest infestation inside fruit orchards. The embedded solution is based on a low-power embedded sensing system along with a Neural Accelerator able to capture and process images inside common pheromone-based traps. Three different ML algorithms have been trained and deployed, highlighting the capabilities of the platform. Moreover, the proposed approach guarantees an extended battery life thanks to the integration of energy harvesting functionalities. Results show how it is possible to automate the task of pest infestation for unlimited time without the farmer's intervention.

Neural Network Training Acceleration With RRAM-Based Hybrid Synapses.

Hardware neural network (HNN) based on analog synapse array excels in accelerating parallel computations. To implement an energy-efficient HNN with high accuracy, high-precision synaptic devices and fully-parallel array operations are essential. However, existing resistive memory (RRAM) devices can represent only a finite number of conductance states. Recently, there have been attempts to compensate device nonidealities using multiple devices per weight. While there is a benefit, it is difficult to apply the existing parallel updating scheme to the synaptic units, which significantly increases updating process’s cost in terms of computation speed, energy, and complexity. Here, we propose an RRAM-based hybrid synaptic unit consisting of a “big” synapse and a “small” synapse, and a related training method. Unlike previous attempts, array-wise fully-parallel learning is possible with our proposed architecture with a simple array selection logic. To experimentally verify the hybrid synapse, we exploit Mo/TiOx RRAM, which shows promising synaptic properties and areal dependency of conductance precision. By realizing the intrinsic gain via proportionally scaled device area, we show that the big and small synapse can be implemented at the device-level without modifications to the operational scheme. Through neural network simulations, we confirm that RRAM-based hybrid synapse with the proposed learning method achieves maximum accuracy of 97 %, comparable to floating-point implementation (97.92%) of the software even with only 50 conductance states in each device. Our results promise training efficiency and inference accuracy by using existing RRAM devices.

Neural Acceleration of Scattering‐Aware Color 3D Printing

AbstractWith the wider availability of full‐color 3D printers, color‐accurate 3D‐print preparation has received increased attention. A key challenge lies in the inherent translucency of commonly used print materials that blurs out details of the color texture. Previous work tries to compensate for these scattering effects through strategic assignment of colored primary materials to printer voxels. To date, the highest‐quality approach uses iterative optimization that relies on computationally expensive Monte Carlo light transport simulation to predict the surface appearance from subsurface scattering within a given print material distribution; that optimization, however, takes in the order of days on a single machine. In our work, we dramatically speed up the process by replacing the light transport simulation with a data‐driven approach. Leveraging a deep neural network to predict the scattering within a highly heterogeneous medium, our method performs around two orders of magnitude faster than Monte Carlo rendering while yielding optimization results of similar quality level. The network is based on an established method from atmospheric cloud rendering, adapted to our domain and extended by a physically motivated weight sharing scheme that substantially reduces the network size. We analyze its performance in an end‐to‐end print preparation pipeline and compare quality and runtime to alternative approaches, and demonstrate its generalization to unseen geometry and material values. This for the first time enables full heterogenous material optimization for 3D‐print preparation within time frames in the order of the actual printing time.

SNAP: An Efficient Sparse Neural Acceleration Processor for Unstructured Sparse Deep Neural Network Inference

Recent developments in deep neural network (DNN) pruning introduces data sparsity to enable deep learning applications to run more efficiently on resourceand energy-constrained hardware platforms. However, these sparse models require specialized hardware structures to exploit the sparsity for storage, latency, and efficiency improvements to the full extent. In this work, we present the sparse neural acceleration processor (SNAP) to exploit unstructured sparsity in DNNs. SNAP uses parallel associative search to discover valid weight (W) and input activation (IA) pairs from compressed, unstructured, sparse W and IA data arrays. The associative search allows SNAP to maintain a 75% average compute utilization. SNAP follows a channel-first dataflow and uses a two-level partial sum (psum) reduction dataflow to eliminate access contention at the output buffer and cut the psum writeback traffic by 22× compared with state-of-the-art DNN accelerator designs. SNAP's psum reduction dataflow can be configured in two modes to support general convolution (CONV) layers, pointwise CONV, and fully connected layers. A prototype SNAP chip is implemented in a 16-nm CMOS technology. The 2.3-mm2 test chip is measured to achieve a peak effectual efficiency of 21.55 TOPS/W (16 b) at 0.55 V and 260 MHz for CONV layers with 10% weight and activation densities. Operating on a pruned ResNet-50 network, the test chip achieves a peak throughput of 90.98 frames/s at 0.80 V and 480 MHz, dissipating 348 mW.

Improving the Performance of Charge Trapping Memtransistor as Synaptic Device by Ti-Doped HfO2

In this work, we improved the performance of germanium (Ge) channel Charge Trapping MemTransistors (CTMTs) as synaptic device by using Ti-doped HfO2 as charge trapping layer (CTL). We manipulated the amount of Ti dopant within the HfO2 CTL to perform the band engineering by varying the Hf/Ti cycle ratio in atomic layer deposition (ALD). The content of Ti was quantified and the energy band structures of the gate stack was constructed with the aid of transmission electron microscope (TEM) images and X-ray photoelectron spectroscopy (XPS) analysis. We then fabricated the charge trapping capacitors and characterized their memory characteristics such as memory windows. By the implementation of amphoteric trap model, thermal activated electron retention model and advanced charge decay model, the trap distribution of the CTL was extracted. Finally, we fabricated the CTMTs with Ti-doped HfO2 as the CTL and characterized their performance as synaptic device such as nonlinearity of depression and potentiation and also conductance on/off ratio. We used NeuroSim simulator with multilayer perceptron and convolutional neural network models to evaluate the pattern recognition accuracy of neural network hardware accelerator using CTMTs as synaptic devices and benchmarked the performance of our CTMT with those of other types of synaptic devices.