Energy-efficient Neural Network Research Articles

Magnetic field-free stochastic computing based on the voltage-controlled magnetic tunnel junction

The stochastic computing (SC) has been proven to be an energy-efficient way to perform neural network. In this study, we propose a field-free voltage-controlled spintronics SC system based on the magnetic tunnel junction (MTJ). We observe a stochastic switching behavior of the MTJ and that it could be controlled by the voltage applied on the device. The voltage-controlled stochastic switching behavior is used to encode numbers ranging from 0 to 1 into a series of random bit-streams in the SC system. Furthermore, the handwritten recognition task is performed on the MTJ-based SC system, achieving a 95% maximum accuracy, which is comparable with the floating-point based neural network. Our work provides inspiration for the energy-efficient neural network systems.

Ferroelectric Domain Wall Delayer and Low-Dropout Regulator.

A switching-type power converter providing an accurate and stable switching output voltage against line/load variations and power supply ripple is mostly complicated in system-on-chip power management integrated circuits (PMICs) within a limited occupation area. Here we fabricated domain wall (DW) nanodevices using an X-cut LiNbO3 thin film on silicon. The domain switching event occurs after a delay time predicted by Merz's law under the applied voltage. But the output current is irrespective of the applied voltage and can be adjusted by conducting wall width as well as input resistance in the circuit. The regulating currents appear repetitively across the volatile interfacial domains between the nanodevice and electrode under intermittently applied voltages. A wall-current-limited domain switching model is developed to explain the phenomenon. The multifunctional DW nanodevices with smaller occupation areas can serve as compact low-dropout regulators in PMICs, time-domain delayers in energy-efficient neural network systems, and on-chip electrostatic discharge protection besides nonvolatile memories and selectors.

Sharing leaky-integrate-and-fire neurons for memory-efficient spiking neural networks.

Spiking Neural Networks (SNNs) have gained increasing attention as energy-efficient neural networks owing to their binary and asynchronous computation. However, their non-linear activation, that is Leaky-Integrate-and-Fire (LIF) neuron, requires additional memory to store a membrane voltage to capture the temporal dynamics of spikes. Although the required memory cost for LIF neurons significantly increases as the input dimension goes larger, a technique to reduce memory for LIF neurons has not been explored so far. To address this, we propose a simple and effective solution, EfficientLIF-Net, which shares the LIF neurons across different layers and channels. Our EfficientLIF-Net achieves comparable accuracy with the standard SNNs while bringing up to ~4.3× forward memory efficiency and ~21.9× backward memory efficiency for LIF neurons. We conduct experiments on various datasets including CIFAR10, CIFAR100, TinyImageNet, ImageNet-100, and N-Caltech101. Furthermore, we show that our approach also offers advantages on Human Activity Recognition (HAR) datasets, which heavily rely on temporal information. The code has been released at https://github.com/Intelligent-Computing-Lab-Yale/EfficientLIF-Net.

Neural architecture search for energy-efficient always-on audio machine learning

Mobile and edge computing devices for always-on classification tasks require energy-efficient neural network architectures. In this paper we present several changes to neural architecture searches that improve the chance of success in practical situations. Our search simultaneously optimizes for network accuracy, energy efficiency and memory usage. We benchmark the performance of our search on real hardware, but since running thousands of tests with real hardware is difficult, we use a random forest model to roughly predict the energy usage of a candidate network. We present a search strategy that uses both Bayesian and regularized evolutionary search with particle swarms, and employs early stopping to reduce the computational burden. Our search, evaluated on a sound event classification dataset based upon AudioSet, results in an order of magnitude less energy per inference and a much smaller memory footprint than our baseline MobileNetV1/V2 implementations while slightly improving task accuracy. We also demonstrate how combining a 2D spectrogram with a convolution with many filters causes a computational bottleneck for audio classification and that alternative approaches reduce the computational burden but sacrifice task accuracy.

A Threshold Implementation-Based Neural Network Accelerator With Power and Electromagnetic Side-Channel Countermeasures

With the recent advancements in machine learning (ML) theory, a lot of energy-efficient neural network (NN) accelerators have been developed. However, their associated side-channel security vulnerabilities pose a major concern. There have been several proof-of-concept attacks demonstrating the extraction of their model parameters and input data. This work introduces a threshold implementation (TI) masking-based NN accelerator that secures model parameters and inputs against power and electromagnetic (EM) side-channel attacks. The 0.159 mm2 demonstration in 28 nm runs at 125 MHz at 0.95 V and limits the area and energy overhead to 64% and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$5.5\times $ </tex-math></inline-formula> , respectively, while demonstrating security even greater than 2M traces. The accelerator also secures model parameters through encryption and the inputs against horizontal power analysis (HPA) attacks.

S[formula omitted]NN: Time step reduction of spiking surrogate gradients for training energy efficient single-step spiking neural networks

As the scales of neural networks increase, techniques that enable them to run with low computational cost and energy efficiency are required. From such demands, various efficient neural network paradigms, such as spiking neural networks (SNNs) or binary neural networks (BNNs), have been proposed. However, they have sticky drawbacks, such as degraded inference accuracy and latency. To solve these problems, we propose a single-step spiking neural network (S3NN), an energy-efficient neural network with low computational cost and high precision. The proposed S3NN processes the information between hidden layers by spikes as SNNs. Nevertheless, it has no temporal dimension so that there is no latency within training and inference phases as BNNs. Thus, the proposed S3NN has a lower computational cost than SNNs that require time-series processing. However, S3NN cannot adopt naïve backpropagation algorithms due to the non-differentiability nature of spikes. We deduce a suitable neuron model by reducing the surrogate gradient for multi-time step SNNs to a single-time step. We experimentally demonstrated that the obtained surrogate gradient allows S3NN to be trained appropriately. We also showed that the proposed S3NN could achieve comparable accuracy to full-precision networks while being highly energy-efficient.

An Energy Efficient Time-Multiplexing Computing-in-Memory Architecture for Edge Intelligence

The growing data volume and complexity of deep neural networks (DNNs) require new architectures to surpass the limitation of the von-Neumann bottleneck, with computing-in-memory (CIM) as a promising direction for implementing energy-efficient neural networks. However, CIM’s peripheral sensing circuits are usually power- and area-hungry components. We propose a time-multiplexing CIM architecture (TM-CIM) based on memristive analog computing to share the peripheral circuits and process one column at a time. The memristor array is arranged in a column-wise manner that avoids wasting power/energy on unselected columns. In addition, digital-to-analog converter (DAC) power and energy efficiency, which turns out to be an even greater overhead than analog-to-digital converter (ADC), can be fine-tuned in TM-CIM for significant improvement. For a 256*256 crossbar array with a typical setting, TM-CIM saves <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$18.4\times $ </tex-math></inline-formula> in energy with 0.136 pJ/MAC efficiency, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$19.9\times $ </tex-math></inline-formula> area for 1T1R case and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$15.9\times $ </tex-math></inline-formula> for 2T2R case. Performance estimation on VGG-16 indicates that TM-CIM can save over <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$16\times $ </tex-math></inline-formula> area. A tradeoff between the chip area, peak power, and latency is also presented, with a proposed scheme to further reduce the latency on VGG-16, without significantly increasing chip area and peak power.

Review and Analysis of Variable Bit-precision MAC Microarchitectures for Energy-efficient AI Computation

In this paper, we review and analyze the state-of-the-art bit-precision scalable multiply-accumulate (MAC) unit microarchitectures. To enable fast and energy-efficient neural network computation, many previous MAC microarchitectures have been explored, but it is hard to select an optimal approach because those works were usually evaluated in different experimental conditions. We first review various variable-bit MAC microarchitectures. Then, we synthesize the variable-bit hardware microarchitectures in a 28 nm technology, and we evaluate the synthesized designs using our custom architecture simulator. Using real benchmarks, we compare various bit-scalable hardware topologies to help researchers choose the most suitable microarchitecture for variable-bit hardware design.

Analysis of electro-chemical RAM synaptic array for energy-efficient weight update

While electro-chemical RAM (ECRAM)-based cross-point synaptic arrays are considered to be promising candidates for energy-efficient neural network computational hardware, array-level analyses to achieve energy-efficient update operations have not yet been performed. In this work, we fabricated CuOx/HfOx/WOx ECRAM arrays and demonstrated linear and symmetrical weight update capabilities in both fully parallel and sequential update operations. Based on the experimental measurements, we showed that the source-drain leakage current (ISD) through the unselected ECRAM cells and resultant energy consumption—which had been neglected thus far—contributed a large portion to the total update energy. We showed that both device engineering to reduce ISD and the selection of an update scheme—for example, column-by-column—that avoided ISD intervention via unselected cells were key to enable energy-efficient neuromorphic computing.

STICKER-IM: A 65 nm Computing-in-Memory NN Processor Using Block-Wise Sparsity Optimization and Inter/Intra-Macro Data Reuse

Computing-in-memory (CIM) is a promising architecture for energy-efficient neural network (NN) processors. Several CIM macros have demonstrated high energy efficiency, while CIM-based system-on-a-chip is not well explored. This work presents a CIM NN processor, named STICKER-IM, which is implemented with sophisticated system integration. Three key innovations are proposed. First, a CIM-friendly block-wise sparsity (BWS) architecture is designed, enabling both activation-sparsity-aware acceleration and weight-sparsity-aware power-saving. Second, an adaptive kernel-/channel-order (KCO) mapping and intra-/inter-macro scheduling strategy is proposed to improve macro utilization and data reuse. Third, an efficient BWS-optimized CIM (BWS-CIM) macro with adaptive power-OFF ADCs is implemented. The STICKER-IM chip was fabricated in 65-nm CMOS technology. Experimental results show 5.8–158-TOPS/W average system energy efficiency on the sparse NN models. The macro/system-level energy efficiency is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$4.23\times / 3.06\times $ </tex-math></inline-formula> higher compared with the state-of-the-art CIM macros and processors.

Saving Energy of RRAM-Based Neural Accelerator Through State-Aware Computing

In-memory computing (IMC) is recognized as one of the most promising architecture solution to realize energy-efficient neural network inference. Amongst many memory technology, resistive RAM (RRAM) is a very attractive device to implement the IMC-based neural network accelerator architecture, which is particularly suitable for power-constrained IoT systems. Due to the nature of low leakage and in-situ computing, the dynamic power consumption of dot-production operations in RRAM crossbars dominates the chip power, especially when applied to low-precision neural networks. This work investigates the correlation between the cell resistance state and the crossbar operation power, and proposes a state-aware RRAM accelerator (SARA) architecture for energy-efficient low-precision neural networks. With the proposed state-aware network training and mapping strategy, crossbars in the RRAM accelerator can perform in a lower power state. Furthermore, we also leverage the proposed RRAM accelerator architecture to reduce the power consumption of high-precision network inference with both single-level or multilevel RRAM. The evaluation results show that for binary neural networks, our design saves 40.53% RRAM computing energy on average over the baseline. For high precision neural networks, the proposed method reduces 11.67% computing energy on average without any accuracy loss.

Design of Binary Convolution Operation Circuit for Binarized Neural Networks Using Single-Flux-Quantum Circuit

We design a binary convolution operation circuit (BCOC) using a single-flux-quantum circuit for high-speed and energy-efficient neural network. The proposed circuit is used for binary convolution operations using a convolution kernel size of 3 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$ \times $</tex-math></inline-formula> 3, which accelerates the forward propagation process of a binary neural network (BNN). We analyze the binary convolution process and propose a bisection method for optimization. The BCOC is designed with a gate-level pipeline architecture and uses the bisection method for reduced number of pipeline stages. Thus, the circuit area of the BCOC is reduced by approximately 50% compared with that of a BCOC without the bisection method. We design the BCOC with 3270 Josephson junctions using a 10 kA/cm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> Nb process. The measurement results show that the BCOC can perform binary convolution operations with a kernel size of 3 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$ \times $</tex-math></inline-formula> 3. Compared to a CMOS circuit, BCOC increases the power efficiency by 3.9 times. In future research, we will build up a library of BNNs based on SFQ circuits to simulate various BNN structures.

An on-chip photonic deep neural network for image classification.

Deep neural networks with applications from computer vision to medical diagnosis1-5 are commonly implemented using clock-based processors6-14, in which computation speed is mainly limited by the clock frequency and the memory access time. In the optical domain, despite advances in photonic computation15-17, the lack of scalable on-chip optical non-linearity and the loss of photonic devices limit the scalability of optical deep networks. Here we reportan integrated end-to-end photonic deep neural network (PDNN) that performs sub-nanosecond image classification through direct processing of the optical waves impinging on the on-chip pixel array as they propagate through layers of neurons. In each neuron, linear computation is performed optically and the non-linear activation function is realized opto-electronically, allowing a classification time of under 570 ps, which is comparable with a single clock cycle of state-of-the-art digital platforms. A uniformly distributed supply light provides the same per-neuron optical output range, allowing scalability to large-scale PDNNs. Two-class and four-class classification of handwritten letters with accuracies higher than 93.8% and 89.8%, respectively, is demonstrated. Direct, clock-less processing of optical data eliminates analogue-to-digital conversion and the requirement for a large memory module, allowing faster and more energyefficient neural networks for the next generations of deep learning systems.

A Neural Network Training Processor With 8-Bit Shared Exponent Bias Floating Point and Multiple-Way Fused Multiply-Add Trees

Recent advances in deep neural networks (DNNs) and machine learning algorithms have induced the demand for services based on machine learning algorithms that require a large number of computations, and specialized hardware ranging from accelerators for data centers to on-device computing systems have been introduced. Low-precision math such as 8-bit integers have been used in neural networks for energy-efficient neural network inference, but training with low-precision numbers without performance degradation have remained to be a challenge. To overcome this challenge, this article presents an 8-bit floating-point neural network training processor for state-of-the-art non-sparse neural networks. As naïve 8-bit floating-point numbers are insufficient for training DNNs robustly, two additional methods are introduced to ensure high-performance DNN training. First, a novel numeric system which we dub as 8-bit floating point with shared exponent bias (FP8-SEB) is introduced. Moreover, multiple-way fused multiply-add (FMA) trees are used in FP8-SEB’s hardware implementation to ensure higher numerical precision and reduced energy. FP8-SEB format combined with multiple-way FMA trees is evaluated under various scenarios to show a trained-from-scratch performance that is close to or even surpasses that of current networks trained with full-precision (FP32). Our silicon-verified DNN training processor utilizes 24-way FMA trees implemented with FP8-SEB math and flexible 2-D routing schemes to show <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.48\times $ </tex-math></inline-formula> higher energy efficiency than prior low-power neural network training processors and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$78.1\times $ </tex-math></inline-formula> lower energy than standard GPUs.

A 28-nm 198.9-TOPS/W Fault-Tolerant Stochastic Computing Neural Network Processor

Previous energy-efficient neural network (NN) processors suffer from bit errors when operating at lower voltages for further power reduction. Stochastic computing (SC) shows great potential due to its low hardware cost and high fault tolerance. Conventionally, limited by the long latency of bitstreams, SC-based NN accelerators adopt a hybrid stochasticbinary architecture, sacrificing fault tolerance and hardware efficiency. This paper proposes a fully SC architecture that maximizes fault tolerance while offering excellent energy and area efficiency. The fabricated 28nm prototype is the first siliconproven SC-based NN processor, realizing an energy efficiency of 198.9 TOPS/W and an area efficiency of 2630 GOPS/mm2 with an accuracy loss reduction of 70%.

A Multiplier-Less Convolutional Neural Network Inference Accelerator for Intelligent Edge Devices

As the demand for neural network operations on edge devices increases, energy-efficient neural network inference solutions become necessary. To this end, this paper proposes a compact 4-bit number format (SD4) for neural network weights. In addition to significantly reducing the amount of neural network data transmission, SD4 also reduces the neural network convolution operation from multiplication and addition (MAC) to only addition. MNIST and CIFAR-10 CNNs with SD4 weights achieve results similar to their FP32-trained counterparts. The difference between the top-1 accuracy of 4-bit ResNet CNN for ImageNet and the baseline FP32 CNN is less than 0.5%. In the hardware design, we have implemented a multiplier-less convolution acceleration circuit. Compared with the 8-bit weight circuit, the power consumption and area of a 4-bit <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3\times 3$ </tex-math></inline-formula> convolution circuit are reduced by nearly 50%. This work also proposes a systematic CNN deployment solution consisting of software CNN training and hardware acceleration. The proposed FPGA-based accelerator for VGG7 image classification achieves a peak throughput of 345.6 GOPS when running at a 100-MHz clock rate. The proposed convolution accelerator’s power consumption and energy efficiency are 1.19W and 289. 5 GOPS/W, respectively. Compared to the CPU implementation of VGG7-128 inference, the multiplier-less acceleration circuit is 4.8 times faster and achieves 384 times higher energy efficiency.

Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks

Inspired by detailed modelling of biological neurons, spiking neural networks (SNNs) are investigated as biologically plausible and high-performance models of neural computation. The sparse and binary communication between spiking neurons potentially enables powerful and energy-efficient neural networks. The performance of SNNs, however, has remained lacking compared with artificial neural networks. Here we demonstrate how an activity-regularizing surrogate gradient combined with recurrent networks of tunable and adaptive spiking neurons yields the state of the art for SNNs on challenging benchmarks in the time domain, such as speech and gesture recognition. This also exceeds the performance of standard classical recurrent neural networks and approaches that of the best modern artificial neural networks. As these SNNs exhibit sparse spiking, we show that they are theoretically one to three orders of magnitude more computationally efficient compared to recurrent neural networks with similar performance. Together, this positions SNNs as an attractive solution for AI hardware implementations. The use of sparse signals in spiking neural networks, modelled on biological neurons, offers in principle a highly efficient approach for artificial neural networks when implemented on neuromorphic hardware, but new training approaches are needed to improve performance. Using a new type of activity-regularizing surrogate gradient for backpropagation combined with recurrent networks of tunable and adaptive spiking neurons, state-of-the-art performance for spiking neural networks is demonstrated on benchmarks in the time domain.

An Approximate GEMM Unit for Energy-Efficient Object Detection.

Edge computing brings artificial intelligence algorithms and graphics processing units closer to data sources, making autonomy and energy-efficient processing vital for their design. Approximate computing has emerged as a popular strategy for energy-efficient circuit design, where the challenge is to achieve the best tradeoff between design efficiency and accuracy. The essential operation in artificial intelligence algorithms is the general matrix multiplication (GEMM) operation comprised of matrix multiplication and accumulation. This paper presents an approximate general matrix multiplication (AGEMM) unit that employs approximate multipliers to perform matrix–matrix operations on four-by-four matrices given in sixteen-bit signed fixed-point format. The synthesis of the proposed AGEMM unit to the 45 nm Nangate Open Cell Library revealed that it consumed only up to 36% of the area and 25% of the energy required by the exact general matrix multiplication unit. The AGEMM unit is ideally suited to convolutional neural networks, which can adapt to the error induced in the computation. We evaluated the AGEMM units’ usability for honeybee detection with the YOLOv4-tiny convolutional neural network. The results implied that we can deploy the AGEMM units in convolutional neural networks without noticeable performance degradation. Moreover, the AGEMM unit’s employment can lead to more area- and energy-efficient convolutional neural network processing, which in turn could prolong sensors’ and edge nodes’ autonomy.

Energy-efficient neural networks with near-threshold processors and hardware accelerators

Hardware for energy-efficient AI has received significant attention over the last years with both start-ups and large corporations creating products that compete at different levels of performance and power consumption. The main objective of this hardware is to offer levels of efficiency and performance that cannot be obtained with general-purpose processors or graphics processing units. In parallel, innovative hardware techniques such as near- and sub-threshold voltage processing have been revisited, capitalizing on the low-power requirements of deploying AI at the network edge. In this paper, we evaluate recent developments in hardware for energy-efficient AI, focusing on inference in embedded systems at the network edge. We then explore a heterogeneous configuration that deploys a neural network that processes multiple independent inputs and deploys convolutional and LSTM (Long Short-Term Memory) layers. This heterogeneous configuration uses two devices with different performance/power characteristics connected with a feedback loop. It obtains energy reductions measured at 75% while simultaneously maintaining the level of inference accuracy.

Efficient Deployment of Spiking Neural Networks on SpiNNaker Neuromorphic Platform

Spiking Neural Networks (SNNs) have emerged as serious competitors of the traditional Convolutional Neural Networks (CNNs), as they unlock new potential of implementing less complex and more energy efficient neural networks. Current deep CNNs can be converted to SNNs for fast deployment on neuromorphic devices, however existing methods do not investigate the impact of hardware-related parameters that directly affect the accuracy of an SNN. In this brief, we target the SpiNNaker neuromorphic platform and we demonstrate a fast exploration framework that effectively decides the configuration of the target board, in order to achieve the highest possible accuracy. Experimental results show that our method reaches 98.85% SNN accuracy on MNIST dataset, while reducing the exploration time by a factor of 3× compared to exhaustive search.