Neural Network Hardware Research Articles

Design and implementation of deep neural network hardware chip and its performance analysis

The artificial neural network (ANN) with a single layer has a limited capacity to process data. Multiple neurons are connected in the human brain, and the actual capacity of the brain lies in the interconnectedness of multiple neurons. As a specified generalization of ANN deep learning makes use of two or more hidden layers, which implies that a greater number of neurons are required to construct the model. A network that has more than one hidden layer, also known as two or more hidden layers, is referred to as a deep neural network, and the process of training such networks is referred to as deep learning. The research article focuses on the design of a multilayer or deep neural network presented for the target field programmable gate array (FPGA) device spartan-6 (xc6stx4-2t9g144) FPGA. The simulation is carried out using Xilinx ISE and ModelSim software. There are two hidden layers in which (2×1) multiplexer blocks are utilized for processing twenty neurons into the output of ten neurons in the first hidden layer and demultiplexers (1×2) and vice versa. The hardware utilization is estimated on FPGA to compute the performance of the deep neural hardware chip based on memory, flip flops, delay, and frequency parameters. The design is scalable and applicable to various FPGA devices, which makes the work novel. FPGA-based neuromorphic hardware acceleration platform with high speed and low power for discrete spike processing on hardware with great real-time performance.

Synchronized stepwise control of firing and learning thresholds in a spiking randomly connected neural network toward hardware implementation

Spiking randomly connected neural network (RNN) hardware is promising as ultimately low power devices for temporal data processing at the edge. Although the potential of RNNs for temporal data processing has been demonstrated, randomness of the network architecture often causes performance degradation. To mitigate such degradation, self-organization mechanism using intrinsic plasticity (IP) and synaptic plasticity (SP) should be implemented in the spiking RNN. Therefore, we propose hardware-oriented models of these functions. To implement the function of IP, a variable firing threshold is introduced to each excitatory neuron in the RNN that changes stepwise in accordance with its activity. We also define other thresholds for SP that synchronize with the firing threshold, which determine the direction of stepwise synaptic update that is executed on receiving a pre-synaptic spike. To discuss the effectiveness of our model, we perform simulations of temporal data learning and anomaly detection using publicly available electrocardiograms (ECGs) with a spiking RNN. We observe that the spiking RNN with our IP and SP models realizes the true positive rate of 1 with the false positive rate being suppressed at 0 successfully, which does not occur otherwise. Furthermore, we find that these thresholds as well as the synaptic weights can be reduced to binary if the RNN architecture is appropriately designed. This contributes to minimization of the circuit of the neuronal system having IP and SP.

Nitrogen-doped carbon quantum dot-decorated In2O3 synaptic transistors for neuromorphic computing

Nitrogen-doped carbon quantum dots (N-CQDs) are promising materials for electronic devices due to their variable bandgap and structural stability. Here, we integrate N-CQDs into In2O3 synaptic transistors with electrolyte gating, resulting in a hybrid structure. The surface functional groups and defects of N-CQDs empower the charge trapping mechanism, permitting controlled conduction and charge regulation, which are crucial for emulating linear and symmetric artificial synaptic devices. Devices incorporating N-CQDs demonstrate enhanced stability and memory characteristics, low energy consumption, consistent retention, and a significant hysteresis window across multiple voltage cycles. Finally, the study emulates biological synapses and cognitive functions, achieving an energy consumption of 10 fJ per synaptic event and a pattern recognition accuracy of 91.2% on the MNIST dataset in hardware neural networks. This work demonstrates the potential of well-manipulating charge trapping in N-CQDs to develop high-performance, nonvolatile synaptic devices.

Thermal Instability Compensation of Synaptic 3D Flash Memory-Based Hardware Neural Networks With Adaptive Read Bias

Thermal Instability Compensation of Synaptic 3D Flash Memory-Based Hardware Neural Networks With Adaptive Read Bias

AI-Powered Neural Network Verification: System Verilog Methodologies for Machine Learning in Hardware

This research focuses on verifying neural network models using System Verilog, with two primary applications: visual edge detection and neuron behavior modeling. In modern chip design, hardware verification plays a crucial role in ensuring that complex neural models perform as expected. A neuron model based on Hubel and Wiesel’s feed-forward network architecture was proposed and tested using integrator and threshold modules implemented in Verilog. The proposed verification methodology employs self-checking test benches, supported by functional coverage and simulation, for comprehensive validation. The results demonstrate efficient verification with high coverage, paving the way for future advancements in hardware neural networks.

Field-free multistate spin–orbit torque devices for programmable image edge recognition circuit

The application of spin–orbit torque (SOT) devices to neuromorphic computing platforms is focused on the development of hardware circuit architectures. However, the inter-device variability, the integration modes of devices and peripheral circuits, and appropriate application scenarios are still unclear, limiting the development of SOT devices in neuromorphic computing. To solve this problem, this paper first proposes a circuit compensation scheme for the difference in resistance values of SOT devices, which solves this variability problem at the circuit level. Moreover, a synergistic scheme with the circuit is developed based on the correspondence between the multistate resistance characteristics of the SOT devices and a convolutional algorithm. To achieve this, a multichannel SOT convolutional kernel circuit architecture is built, which implements an image edge recognition application. Finally, based on a simulation model, an image edge recognition hardware circuit based on our CoPt-SOT devices is implemented, which is capable of performing image edge recognition with an accuracy of 96.33%. This scheme provides technical support and development prospects for SOT devices in neural network hardware applications.

Design of Yolov4-Tiny convolutional neural network hardware accelerator based on FPGA

Design of Yolov4-Tiny convolutional neural network hardware accelerator based on FPGA

Study of Soft Errors in Spiking Neural Network Hardware

The problem of soft errors in the hardware implementation of Spiking Neural Networks (SNN) has always been a challenge. SNNs, unlike traditional deep learning networks, simulate the temporal dynamic behavior of biological neurons and emit spikes when a specific threshold is reached. In recent years, the hardware implementation of SNNs has shown great potential in performing efficient and low-power tasks, but as technology nodes shrink and integrated circuit complexity increases, soft errors become a key challenge. To this end, we propose a novel approach based on input and weight analysis, through specific algorithms at the hardware level, to significantly reduce the probability of soft errors affecting the results. In terms of training accuracy, our method maintains a high accuracy under various voltage fluctuation conditions, demonstrating its superior robustness.

Unlocking Neuromorphic Vision: Advancements in IGZO-Based Optoelectronic Memristors with Visible Range Sensitivity.

Optoelectronic memristors based on amorphous oxide semiconductors (AOSs) are promising devices for the development of spiking neural network (SNN) hardware in neuromorphic vision sensors. In such devices, the conductance state can be controlled by both optical and electrical stimuli, while the typical persistent photoconductivity (PPC) of AOS materials can be used to emulate synaptic functions. However, due to the large band gap of these materials, sensitivity to visible light (red/green/blue) is difficult to accomplish, which hinders applications requiring color discrimination. In this work, we report a 4 μm2 hydrogen-doped (H-doped) indium-gallium-zinc oxide (IGZO) optoelectronic memristor that emulates all of the important rules of SNNs such as short- to long-term memory transition (STM-LTM), paired-pulse facilitation (PPF), spike-time-dependent plasticity (STDP), and learning and forgetting capabilities. By the incorporation of hydrogen gas in the sputtering deposition of IGZO, visible sensitivity was achieved for green and blue wavelengths. Additionally, extremely high light/dark ratios of 179, 93, and 12 are demonstrated for wavelengths of 365, 405, and 505 nm, respectively, due to hydrogen-induced subgap states and device miniaturization. Therefore, the proposed device shows remarkable potential for integration with the pixel circuits of IGZO-based displays with extreme resolution for a true intelligent self-processing display.

Energy-efficient neural network using an anisotropy field gradient-based self-resetting neuron and meander synapse

Neuromorphic computing (NC) is considered a potential solution for energy-efficient artificial intelligence applications. The development of reliable neural network (NN) hardware with low energy and area footprints plays a crucial role in realizing NC. Even though neurons and synapses have already been investigated using a variety of spintronic devices, the research is still in the primitive stages. Particularly, there is not much experimental research on the self-reset (and leaky) aspect(s) of domain wall (DW) device-based neurons. Here, we have demonstrated an energy-efficient NN using a spintronic DW device-based neuron with self-reset (leaky) and integrate-and-fire functions. An “anisotropy field gradient” provides the self-resetting behavior of auto-leaky, integrate, and fire neurons. The leaky property of the neuron was experimentally demonstrated using a voltage-assisted modification of the anisotropy field. A synapse with a meander wire configuration was used to achieve multiple-resistance states corresponding to the DW position and controlled pinning of the DW. The NN showed an energy efficiency of 0.189 nJ/image/epoch while achieving an accuracy of 92.4%. This study provides a fresh path for developing more energy-efficient DW-based NN systems.

AMED: Automatic Mixed-Precision Quantization for Edge Devices

Quantized neural networks are well known for reducing the latency, power consumption, and model size without significant harm to the performance. This makes them highly appropriate for systems with limited resources and low power capacity. Mixed-precision quantization offers better utilization of customized hardware that supports arithmetic operations at different bitwidths. Quantization methods either aim to minimize the compression loss given a desired reduction or optimize a dependent variable for a specified property of the model (such as FLOPs or model size); both make the performance inefficient when deployed on specific hardware, but more importantly, quantization methods assume that the loss manifold holds a global minimum for a quantized model that copes with the global minimum of the full precision counterpart. Challenging this assumption, we argue that the optimal minimum changes as the precision changes, and thus, it is better to look at quantization as a random process, placing the foundation for a different approach to quantize neural networks, which, during the training procedure, quantizes the model to a different precision, looks at the bit allocation as a Markov Decision Process, and then, finds an optimal bitwidth allocation for measuring specified behaviors on a specific device via direct signals from the particular hardware architecture. By doing so, we avoid the basic assumption that the loss behaves the same way for a quantized model. Automatic Mixed-Precision Quantization for Edge Devices (dubbed AMED) demonstrates its superiority over current state-of-the-art schemes in terms of the trade-off between neural network accuracy and hardware efficiency, backed by a comprehensive evaluation.

Review of Advanced Methods in Hardware Acceleration for Deep Neural Networks

Abstract: Convolutional neural networks have become very efficient in performing tasks like Object Detection providing human like accuracy. However, their practical implementation needs significant hardware resources and memory bandwidth. In recent past a lot of research is being carried out for achieving higher efficiency in implementing such neural networks in hardware. We talk about FPGAs for hardware implementation due to their flexibility for customisation for such neural network architectures. In this paper we will discuss the metrics for efficient hardware accelerator and general methods available for achieving an efficient design. Further, we will discuss the actual methods used by recent research for implementation of deep neural networks particularly for object detection related applications. These methods range from actual ASIC design like TPUs [1] for on chip acceleration, state of the art open source designs like Gemini to methods like hardware reuse, re-configurable nodes and approximation in computations as a trade-off between speed and accuracy. This paper will be a valuable summary for the researchers starting in the field of hardware accelerators design for neural networks

Optimization research of handwritten digit recognition algorithm based on hardware neural networks

In the modern era of digitization, the widespread application of handwritten numerical data has led to an exponential increase in data volume, necessitating efficient real-time recognition and processing systems. This project is grounded in the foundational structure of neural networks, with a specific focus on integrating these networks into hardware through Very-Large-Scale Integration (VLSI). The primary objective is to achieve faster processing speeds, lower power consumption, and real-time operations. The central emphasis of this study is the design and implementation of a hardware neural network system tailored specifically for handwritten digit recognition. Drawing from neural network theory, we explore the construction of a hardware-based handwritten digit classifier. The fundamental model employed is the single-layer perceptron neuron model. Upon receiving 28x28 pixel grayscale images, the image classifier utilizes pre-trained weights, activation functions, and a maximum selector to compare and output recognition results for numerical digits. The concrete implementation is realized using the Verilog hardware description language, coupled with algorithm optimization strategies to enhance performance and efficiency. This research endeavor aims to provide an effective hardware solution for real-time handwritten digit recognition in the digital age.

Fully neuromorphic vision and control for autonomous drone flight.

Biological sensing and processing is asynchronous and sparse, leading to low-latency and energy-efficient perception and action. In robotics, neuromorphic hardware for event-based vision and spiking neural networks promises to exhibit similar characteristics. However, robotic implementations have been limited to basic tasks with low-dimensional sensory inputs and motor actions because of the restricted network size in current embedded neuromorphic processors and the difficulties of training spiking neural networks. Here, we present a fully neuromorphic vision-to-control pipeline for controlling a flying drone. Specifically, we trained a spiking neural network that accepts raw event-based camera data and outputs low-level control actions for performing autonomous vision-based flight. The vision part of the network, consisting of five layers and 28,800 neurons, maps incoming raw events to ego-motion estimates and was trained with self-supervised learning on real event data. The control part consists of a single decoding layer and was learned with an evolutionary algorithm in a drone simulator. Robotic experiments show a successful sim-to-real transfer of the fully learned neuromorphic pipeline. The drone could accurately control its ego-motion, allowing for hovering, landing, and maneuvering sideways-even while yawing at the same time. The neuromorphic pipeline runs on board on Intel's Loihi neuromorphic processor with an execution frequency of 200 hertz, consuming 0.94 watt of idle power and a mere additional 7 to 12 milliwatts when running the network. These results illustrate the potential of neuromorphic sensing and processing for enabling insect-sized intelligent robots.

A logic device based on memristor-diode crossbar and CMOS periphery as spike router for hardware neural network

A programmable logic device based on a memristor-diode crossbar and CMOS logic has been developed. The crossbar implements NAND logic gates using memristor ratioed logic and CMOS inverters. The digitally controlled peripheral circuit provides digital signals transmission and allows modification and evaluation memristor states in the crossbar. The proposed logic device circuit requires fewer transistors than known analogues and less area on the chip.The maximum size of the crossbar in a logic device is estimated by numerical simulation at the level of electrical circuits. The limited size is caused by the degradation of the logic levels voltages in the memristor-diode crossbar. The operability of peripheral circuits as part of a complete electrical circuit of a logic device is demonstrated during the simulation of the execution of logical operations, the processes of modification and evaluation states of individual memristors.

Field Induced Off‐State Instability in InGaZnO Thin‐Film Transistor and its Impact on Synaptic Circuits

AbstractCharge storage synaptic circuits employing InGaZnO thin‐film transistors (IGZO TFTs) and capacitors are a promising candidate for on‐chip trainable neural network hardware accelerators. However, IGZO TFTs often exhibit bias instability. For synaptic memory applications, the programming transistors are predominantly exposed to asymmetric off‐state biases, and a unique field‐dependent on‐current reduction under off‐scenario is observed which may result in programming current variation. Further examination of the phenomenon is conducted with transmission line‐like method and degradation recovery tests, and current reduction can be attributed to contact resistance increase by charge trapping in the source and drain electrode and the channel region. The current decrease is subsequently formulated with a stretched exponential model with bias‐dependent parameters for quantitative circuit analysis under off‐state degradation. A neural network hardware acceleration simulator is utilized to assess the complicated impact the off‐state current degradation could instigate on on‐chip trainable IGZO TFT‐based synapse arrays. The simulation results generally demonstrate deteriorated training accuracy with aggravated off‐state instability, and the accuracy trend is elucidated from the perspective of weight symmetry point.

Compact eternal diffractive neural network chip for extreme environments

Artificial intelligence applications in extreme environments place high demands on hardware robustness, power consumption, and speed. Recently, diffractive neural networks have demonstrated superb advantages in high-throughput light-speed reasoning. However, the robustness and lifetime of existing diffractive neural networks cannot be guaranteed, severely limiting their compactness and long-term inference accuracy. Here, we have developed a millimeter-scale and robust bilayer-integrated diffractive neural network chip with virtually unlimited lifetime for optical inference. The two diffractive layers with binary phase modulation were engraved on both sides of a quartz wafer. Optical inference of handwritten digital recognition was demonstrated. The results showed that the chip achieved 82% recognition accuracy for ten types of digits. Moreover, the chip demonstrated high-performance stability at high temperatures. The room-temperature lifetime was estimated to be 1.84×1023 trillion years. Our chip satisfies the requirements for diffractive neural network hardware with high robustness, making it suitable for use in extreme environments.

Toward Practical Superconducting Accelerators for Machine Learning Using U-SFQ

Most popular superconducting circuits operate on information carried by ps-wide, μV-tall, single flux quantum (SFQ) pulses. These circuits can operate at frequencies of hundreds of GHz with orders of magnitude lower switching energy than complementary-metal-oxide-semiconductors (CMOS). However, under the stringent area constraints of modern superconductor technologies, fully-fledged, CMOS-inspired superconducting architectures cannot be fabricated at large scales. Unary SFQ (U-SFQ) is an alternative computing paradigm that can address these area constraints. In U-SFQ, information is mapped to a combination of streams of SFQ pulses and in the temporal domain. In this work, we extend U-SFQ to introduce novel building blocks such as a multiplier and an accumulator. These blocks reduce area and power consumption by 2\(\times\)and 4\(\times\)compared with previously proposed U-SFQ building blocks and yield at least 97% area savings compared with binary approaches. Using these multiplier and adder, we propose a U-SFQ Convolutional Neural Network (CNN) hardware accelerator capable of comparable peak performance with state-of-the-art superconducting binary approach (B-SFQ) in 32\(\times\)less area. CNNs can operate with 5–8 bits of resolution with no significant degradation in classification accuracy. For 5 bits of resolution, our proposed accelerator yields 5\(\times\)to 63\(\times\)better performance than CMOS and 15\(\times\)to 173\(\times\)better area efficiency than B-SFQ.

Field‐Free Memristive Spin–Orbit Torque Switching in A1 CoPt Single Layer for Image Edge Detection

AbstractWhile spin–orbit torque (SOT) devices are extensively investigated due to their potential for use in neural network computation, it remains challenging to explore the hardware for neural networks. In this paper, the field‐free memristive SOT switching of the CoPt single layer is used to propose a neuromorphic hardware circuit for detecting edges in images. Owing to its threefold symmetry of inversion, the polarity of SOT switching can be reversed by rotating the current by 60°. Moreover, the process of current‐induced SOT switching exhibits stable multi‐state magnetic switching behavior, and can be controllably tuned by using the pulse of the current. As the slope of the applied ramp pulse current increased, the wave of the anomalous Hall resistance changed from a curve with normally memristive property to trigonometric, and finally to cosine. The design of the hardware circuit for a single SOT device is subsequently formulated to detect the edges in images. The results of experiments verified the capability of this device to detect the edge lines in images with high accuracy, which confirms its potential for use in the hardware of neuromorphic computing platforms. The work here provides guidance for the application of SOT‐based devices to neuromorphic hardware.

Robust compression and detection of epileptiform patterns in ECoG using a real-time spiking neural network hardware framework

Interictal Epileptiform Discharges (IED) and High Frequency Oscillations (HFO) in intraoperative electrocorticography (ECoG) may guide the surgeon by delineating the epileptogenic zone. We designed a modular spiking neural network (SNN) in a mixed-signal neuromorphic device to process the ECoG in real-time. We exploit the variability of the inhomogeneous silicon neurons to achieve efficient sparse and decorrelated temporal signal encoding. We interface the full-custom SNN device to the BCI2000 real-time framework and configure the setup to detect HFO and IED co-occurring with HFO (IED-HFO). We validate the setup on pre-recorded data and obtain HFO rates that are concordant with a previously validated offline algorithm (Spearman’s ρ = 0.75, p = 1e-4), achieving the same postsurgical seizure freedom predictions for all patients. In a remote on-line analysis, intraoperative ECoG recorded in Utrecht was compressed and transferred to Zurich for SNN processing and successful IED-HFO detection in real-time. These results further demonstrate how automated remote real-time detection may enable the use of HFO in clinical practice.