Hardware Implementation Of Neural Networks Research Articles

Hybrid CMOS-Memristor synapse circuits for implementing Ca ion-based plasticity model

Neuromorphic computing research is being actively pursued to address the challenges posed by the need for energy-efficient processing of big data. One of the promising approaches to tackle the challenges is the hardware implementation of spiking neural networks (SNNs) with bio-plausible learning rules. Numerous research works have been done to implement the SNN hardware with different synaptic plasticity rules to emulate human brain operations. While a standard spike-timing-dependent-plasticity (STDP) rule is emulated in many SNN hardware, the various STDP rules found in the biological brain have rarely been implemented in hardware. This study proposes a CMOS-memristor hybrid synapse circuit for the hardware implementation of a Ca ion-based plasticity model to emulate the various STDP curves. The memristor was adopted as a memory device in the CMOS synapse circuit because memristors have been identified as promising candidates for analog non-volatile memory devices in terms of energy efficiency and scalability. The circuit design was divided into four sub-blocks based on biological behavior, exploiting the non-volatile and analog state properties of memristors. The circuit was designed to vary weights using an H-bridge circuit structure and PWM modulation. The various STDP curves have been emulated in one CMOS-memristor hybrid circuit, and furthermore a simple neural network operation was demonstrated for associative learning such as Pavlovian conditioning. The proposed circuit is expected to facilitate large-scale operations for neuromorphic computing through its scale-up.

Study of Soft Errors in Spiking Neural Network Hardware

The problem of soft errors in the hardware implementation of Spiking Neural Networks (SNN) has always been a challenge. SNNs, unlike traditional deep learning networks, simulate the temporal dynamic behavior of biological neurons and emit spikes when a specific threshold is reached. In recent years, the hardware implementation of SNNs has shown great potential in performing efficient and low-power tasks, but as technology nodes shrink and integrated circuit complexity increases, soft errors become a key challenge. To this end, we propose a novel approach based on input and weight analysis, through specific algorithms at the hardware level, to significantly reduce the probability of soft errors affecting the results. In terms of training accuracy, our method maintains a high accuracy under various voltage fluctuation conditions, demonstrating its superior robustness.

Recent Trends in Application of Memristor in Neuromorphic Computing: A Review

Abstract: Recently memristors have emerged as a form of nonvolatile memory that is based on the principle of ion transport in solid electrolytes under the impact of an external electric field. It is perceived as one of the key elements to building next-generation computing systems owing to its peculiar resistive switching characteristics. The switching mechanism in a memristor is mainly governed by filamentary conduction. Further, it can be employed as a memory as well as a logic element, which makes it an ideal candidate for building innovative computer architecture. Moreover, it is capable of mimicking the characteristics of biological synapses, which makes it an ideal candidate for developing a Neuromorphic system. In this review to begin with the switching mechanism of the memristor, primarily focusing on filamentary conduction, is discussed. Few SPICE models of memristor are reviewed, and their critical comparison is performed, which are widely used to build computing systems. An in-depth study on the various crossbar memory architecture augmented with memristors is reviewed. Finally, the application of memristors in neuromorphic computing and hardware implementation of Artificial Neural Networks (ANN) employing memristors is discussed.

Dynamics of heterogeneous Hopfield neural network with adaptive activation function based on memristor

Memristor and activation function are two important nonlinear factors of the memristive Hopfield neural network. The effects of different memristors on the dynamics of Hopfield neural networks have been studied by many researchers. However, less attention has been paid to the activation function. In this paper, we present a heterogeneous memristive Hopfield neural network with neurons using different activation functions. The activation functions include fixed activation functions and an adaptive activation function, where the adaptive activation function is based on a memristor. The theoretical and experimental study of the neural network’s dynamics has been conducted using phase portraits, bifurcation diagrams, and Lyapunov exponents spectras. Numerical results show that complex dynamical behaviors such as multi-scroll chaos, transient chaos, state jumps and multi-type coexisting attractors can be observed in the heterogeneous memristive Hopfield neural network. In addition, the hardware implementation of memristive Hopfield neural network with adaptive activation function is designed and verified. The experimental results are in good agreement with those obtained using numerical simulations.

Energy Complexity of Convolutional Neural Networks.

The energy efficiency of hardware implementations of convolutional neural networks (CNNs) is critical to their widespread deployment in low-power mobile devices. Recently, a number of methods have been proposed for providing energy-optimal mappings of CNNs onto diverse hardware accelerators. Their estimated energy consumption is related to specific implementation details and hardware parameters, which does not allow for machine-independent exploration of CNN energy measures. In this letter, we introduce a simplified theoretical energy complexity model for CNNs, based on only a two-level memory hierarchy that captures asymptotically all important sources of energy consumption for different CNN hardware implementations. In this model, we derive a simple energy lower bound and calculate the energy complexity of evaluating a CNN layer for two common data flows, providing corresponding upper bounds. According to statistical tests, the theoretical energy upper and lower bounds we present fit asymptotically very well with the real energy consumption of CNN implementations on the Simba and Eyeriss hardware platforms, estimated by the Timeloop/Accelergy program, which validates the proposed energy complexity model for CNNs.

Dynamic analysis and hardware implementation of multi-scroll Hopfield neural networks with three different memristor synapses

Dynamic analysis and hardware implementation of multi-scroll Hopfield neural networks with three different memristor synapses

Impact of white noise in artificial neural networks trained for classification: Performance and noise mitigation strategies.

In recent years, the hardware implementation of neural networks, leveraging physical coupling and analog neurons has substantially increased in relevance. Such nonlinear and complex physical networks provide significant advantages in speed and energy efficiency, but are potentially susceptible to internal noise when compared to digital emulations of such networks. In this work, we consider how additive and multiplicative Gaussian white noise on the neuronal level can affect the accuracy of the network when applied for specific tasks and including a softmax function in the readout layer. We adapt several noise reduction techniques to the essential setting of classification tasks, which represent a large fraction of neural network computing. We find that these adjusted concepts are highly effective in mitigating the detrimental impact of noise.

Quantization-Based Optimization Algorithm for Hardware Implementation of Convolution Neural Networks

Convolutional neural networks (CNNs) have demonstrated remarkable performance in many areas but require significant computation and storage resources. Quantization is an effective method to reduce CNN complexity and implementation. The main research objective is to develop a scalable quantization algorithm for CNN hardware design and model the performance metrics for the purpose of CNN implementation in resource-constrained devices (RCDs) and optimizing layers in deep neural networks (DNNs). The algorithm novelty is based on blending two quantization techniques to perform full model quantization with optimum accuracy, and without additional neurons. The algorithm is applied to a selected CNN model and implemented on an FPGA. Implementing CNN using broad data is not possible due to capacity issues. With the proposed quantization algorithm, we succeeded in implementing the model on the FPGA using 16-, 12-, and 8-bit quantization. Compared to the 16-bit design, the 8-bit design offers a 44% decrease in resource utilization, and achieves power and energy reductions of 41% and 42%, respectively. Models show that trading off one quantization bit yields savings of approximately 5.4K LUTs, 4% logic utilization, 46.9 mW power, and 147 μJ energy. The models were also used to estimate performance metrics for a sample DNN design.

Multilayer Neural Network Hardware Designs for Faster Convolutional Processing

The demand for efficient hardware implementations of convolutional neural networks (CNNs) has surged with the proliferation of deep learning applications in various domains such as computer vision, natural language processing, and autonomous systems. Convolutional layers, being the fundamental building blocks of CNNs, are computationally intensive, requiring optimized hardware architectures for real-time inference tasks. This paper presents novel hardware designs targeting faster convolutional processing within multilayer neural networks. We propose a multi-faceted approach that leverages parallelism, pipelining, and hardware acceleration techniques to enhance the efficiency of convolution operations. Our design optimally exploits the inherent data-level and model-level parallelism present in CNNs to achieve high throughput while minimizing latency.

Voltage Scaled Low Power DNN Accelerator Design on Reconfigurable Platform

The exponential emergence of Field-Programmable Gate Arrays (FPGAs) has accelerated research on hardware implementation of Deep Neural Networks (DNNs). Among all DNN processors, domain-specific architectures such as Google’s Tensor Processor Unit (TPU) have outperformed conventional GPUs (Graphics Processing Units) and CPUs (Central Processing Units). However, implementing low-power TPUs in reconfigurable hardware remains a challenge in this field. Voltage scaling, a popular approach for energy savings, can be challenging in FPGAs, as it may lead to timing failures if not implemented appropriately. This work presents an ultra-low-power FPGA implementation of a TPU for edge applications. We divide the systolic array of a TPU into different FPGA partitions based on the minimum slack value of different design paths of Multiplier Accumulators (MACs). Each partition uses different near-threshold (NTC) biasing voltages to run its FPGA cores. The biasing voltage for each partition is roughly calculated by the proposed static schemes. However, further calibration of biasing voltage is performed by the proposed runtime scheme. To overcome the timing failure caused by NTC, the MACs with higher minimum slack are placed in lower-voltage partitions, while the MACs with lower minimum slack paths are placed in higher-voltage partitions. The proposed architecture is implemented in a commercial platform, namely Vivado with Xilinx Artix-7 FPGA and academic platform VTR with 22 nm, 45 nm and 130 nm FPGAs. Any timing error caused by NTC can be caught by the Razor flipflop used in each MAC. The proposed voltage-scaled, partitioned systolic array can save 3.1% to 11.6% of dynamic power in Vivado and VTR tools, respectively, depending on the FPGA technology, partition size, number of partitions and biasing voltages. The normalized performance and accuracy of benchmark models running on our low-power TPU are very competitive compared to existing literature.

Multi-bit MRAM based high performance neuromorphic accelerator for image classification

Binary neural networks (BNNs) are the most efficient solution to bridge the design gap of the hardware implementation of neural networks in a resource-constrained environment. Spintronics is a prominent technology among emerging fields for next-generation on-chip non-volatile memory. Spin transfer torque (STT) and spin-orbit torque (SOT) based magnetic random-access memory (MRAM) offer non-volatility and negligible static power. Over the last few years, STT and SOT-based multilevel spintronic memories have emerged as a promising solution to attain high storage density. This paper presents the operation principle and performance evaluation of spintronics-based single-bit STT and SOT MRAM, dual-level cells, three-level cells (TLCs), and four-level cells. Further, multi-layer perceptron architectures have been utilized to perform MNIST image classification with these multilevel devices. The performance of the complete system level consisting of crossbar arrays with various MRAM bit cells in terms of area, energy, and latency is evaluated. The throughput efficiency of the BNN accelerator using TLCs is 26.6X, and 3.61X higher than conventional single-bit STT-MRAM, and SOT-MRAM respectively.

Improved the voltage hysteresis window and endurance in NbOx threshold device by Ti doping

The artificial neuron based on the NbOx threshold device has exhibited great potential for the hardware implementation of neural networks. Improving the margin of frequency tunability and endurance contributes to expanding its application scenarios effectively. In this work, we prepare NbOx films with different Ti doping contents by magnetron sputtering, and the low concentration of Ti-doped device demonstrates high performance with a sufficient and reliable voltage hysteresis window. Due to the small amount of Ti doping, the Nb–Nb bonding robustness is enhanced and the oxygen vacancy formation energy is reduced, as confirmed by the first principles calculations. The former facilitates broadening the voltage hysteresis window and the latter improves the endurance. Furthermore, the margin of the neuron achieved with this device is improved experimentally. This work confirms a promising approach to designing specialized neurons, advancing the development of full memristive neural networks.

A Real-time P-SFA hardware implementation of Deep Neural Networks using FPGA

Machine Learning (ML) algorithms, specifically Artificial Neural Networks (ANNs), have proved their effectiveness in solving complex problems in many different applications and multiple fields. This paper focuses on optimizing the activation function (AF) block of the NN hardware architecture. The AF block used is based on a probability-based sigmoid function approximation block (P-SFA) combined with a novel real-time probability module (PRT) that calculates the probability of the input data. The proposed NN design aims to use the least amount of hardware resources and area while maintaining a high recognition accuracy. The proposed AF module in this work consists of two P-SFA blocks and the PRT component. The architecture proposed for implementing NNs is evaluated on Field Programmable Gate Arrays (FPGAs). The proposed design has achieved a recognition accuracy of 97.84 % on a 6-layer Deep Neural Network (DNN) for the MNIST dataset and a recognition accuracy of 88.58% on a 6-layer DNN for the FMNIST dataset. The proposed AF module has a total area of 1136 LUTs and 327 FFs, a logical critical path delay of 8.853 ns. The power consumption of the P-SFA block is 6 mW and the PRT block is 5 mW.

Monolithically Integrated Complementary Ferroelectric FET XNOR Synapse for the Binary Neural Network.

Neuromorphic computing, which mimics the structure and principles of the human brain, has the potential to facilitate the hardware implementation of next-generation artificial intelligence systems and process large amounts of data with very low power consumption. Among them, the XNOR synapse-based Binary Neural Network (BNN) has been attracting attention due to its compact neural network parameter size and low hardware cost. The previous XNOR synapse has drawbacks, such as a trade-off between cell density and accuracy. In this work, we show nonvolatile XNOR synapses with high density and accuracy using a monolithically stacked complementary ferroelectric field-effect transistor (C-FeFET) composed of a p-type Si MFMIS-FeFET at the bottom and a 3D stackable n-type Al:IZTO MFS-FeTFT, achieving 60F2 per cell (2C-FeFET). For adjusting the threshold voltage and improving the switching speed (100 ns) of n-type ferroelectric TFT, we employed a dual-gate configuration and a unique operation scheme, making it comparable to those of Si-based FeFETs. We performed array-level simulation with a 512 × 512 subarray size and a 3-bit flash ADC, demonstrating that the image recognition accuracies using the MNIST and CIFAR-10 data sets were increased by 3.17 and 14.07%, respectively, in comparison to other nonvolatile XNOR synapses. In addition, we performed system-level analysis on a 512 × 512 XNOR C-FeFET, exhibiting an outstanding throughput of 717.37 GOPS and an energy efficiency of 196.7 TOPS/W. We expect that our approach would contribute to the high-density memory systems, logic-in-memory technology, and hardware implementation of neural networks.

An LSTM-based Neural Network Wearable System for Blood Glucose Prediction in People with Diabetes.

This article proposes the first hardware implemen-tation of a low-power LSTM neural network targeting a wearable medical device designed to predict blood glucose at a 30-minute horizon. This work aims to reduce energy consumption by propos-ing new activation functions that target hardware implementation. On top of this proposal, we also prove there is room for improve-ment in energy consumption by applying neural network optimiza-tions at the algorithmic, such as quantization, and architecture level, LSTM hyperparameters, that consider the target hardware. To validate our proposal, we devise an optimized version of the neural network aimed to be wearable and, therefore, to reduce its energy consumption while preserving its accuracy as much as possible. The hardware is implemented on a Xilinx Virtex-7 FPGA VC707 Evaluation Kit. It is compared with (i) a faithful design of the original neural network implemented on the same evaluation kit, (ii) three state-of-the-art LSTM-based FPGA implementations, and (iii) software implementations running in cutting-edge smartphones:OnePlus NordTM and an Apple iPhone 13 ProTM with artificial in-telligence hardware accelerators. Our proposal consumes between ×1020 and ×7 less energy than the software implementations, being the most efficient system compared to the smartphones. On the other hand, its energy efficiency, measured in GFLOP/J, is between ×2.84 and ×7.82 greater than other state-of-the-art LSTM implementations, proving to be the most suitable implementation for a wearable system for blood glucose prediction.

On the Design of Power Attack Immune Spintronic Associative Memory

The growing utilization of neural networks has led to a heightened focus on the hardware implementation of such networks. Security concerns associated with these implementations pose a significant challenge in this regard. Among these problems, the vulnerability of these networks against side-channel attacks such as power attacks can be mentioned. This paper presents a technique to enhance the resilience of hardware implementations of neural networks, particularly Hopfield neural networks, to mitigate the risks posed by power attacks. In addition to the fact that the proposed method makes it impossible to attack the network, it also reduces the power consumption of the entire circuit by reducing the leakage currents. The simulation results demonstrate that the proposed approach also achieves about a 10% reduction in energy consumption while concurrently improving the accuracy of the implemented associative memory by 1.1%.

An Area-Efficient Integrate-and-Fire Neuron Circuit with Enhanced Robustness against Synapse Variability in Hardware Neural Network

Neuron circuits are the fundamental building blocks in the modern neuromorphic system. Designing compact and low-power neuron circuits can significantly improve the overall area and energy efficiencies of a neuromorphic chip architecture. Here, practical neuron circuits must overcome the variations arising from nonideal behaviors of synaptic devices, such as stuck-at-fault and conductance deviation. In this study, a compact leaky integrate-and-fire neuron circuit has been designed, with resilience to synaptic device state variations, for hardware implementation of spiking neural networks (SNNs). The proposed neuron circuit is simulated on the 0.35-μm Si complementary metal-oxide-semiconductor technology node by a series of circuit simulations based on HSPICE. The proposed circuit occupies a reduced area and exhibits low power consumption (14.7 µW per spike). Furthermore, the optimized circuit design results in a high degree of tolerance toward input-current variations arising from conductance-state variations in the synapse array. Hence, the proposed neuron circuit would be capable of substantially improving the area efficiency and reliability in the realization of the hardware-oriented SNN architectures.

Neuroelectronics as neuromorphic and neurohybryd systems enabled by memristive technology

Due to its distinctive capability to replicate vital functions of synapses and neurons, memristive devices and arrays enable both the hardware implementation of neural networks and a significant advancement towards integrating artificial electronic and biological systems to address pressing issues in artificial intelligence (AI), robotics, and medicine. This research area is still nascent and can be viewed as part of the broader field of neuroelectronics. The aforementioned is a fusion of analog and digital solutions used for diverse computational tasks inspired by biology. Notably, analog neuromorphic systems that employ memristive components are distinctive features of this arena and they can significantly enhance throughput and energy efficiency in contrast to existing AI accelerators. Designing neuromorphic systems grounded on this fresh component base mandates coordinated and interdisciplinary research and development. The foundation of this scientific and technological field lies in the cross-cutting technology of memristive devices and circuits. This technology enables the development of a novel brain-like information and computing system infrastructure that can be applied in a diverse range of fields. Current perspectives include the seamless integration of memristive devices and arrays with CMOS circuits, and the co-optimization of materials, devices, and architectures to create prototypes for computing and information systems. These systems replicate computational features present in biological neural networks capable of solving cognitive tasks that are typically either intractable by traditional AI or highly time-consuming. Neuroelectronic solutions can integrate with the brain or living neuronal cultures to form neurohybrid systems. In this presentation, we discuss two distinct strategies for connecting memristive systems with biological neural networks both in vitro and in vivo. These include a perceptron using an array of programmable memristive weights represented by metal-oxide resistive-switching devices and a methodology leveraging memristive stochastic plasticity and neural synchrony, which is part of the brain’s spiking architecture. Finally, the concept of a memristive neurohybrid chip is presented to create a compact, multifunctional, bidirectional interface between biological neural networks and memristive electronics, combined with microelectrode and microfluidic systems on a single chip. The technological advancements in component base and the development of memristor-based neuroelectronic systems will diversify hardware for the continuous evolution and mass application of artificial intelligence technologies. This will enable the creation of hybrid intelligence based on the symbiosis of artificial and biological neural networks and allow for the establishment of novel tasks at an unprecedented level.

A reservoir computing system with volatile and non-volatile organic memristors as a promising hardware architecture

In recent years, many scientific groups have been working on hardware implementation of the artificial neural networks to approach the computational efficiency of their biological counterpart. Memristors may play the role of synapses in such networks [1]. Varieties of memristive structures and materials have already been tested in different neural network architectures, but still no memristor is considered ideal for hardware synapse implementation [1]. One of the most significant problems is the presence of inherent stochasticity distinctive for all memristive devices, which complicates the training of the neural networks [1]. Several approaches were proposed to partially mitigate this problem, e.g., a reservoir computing system (RCS) [2] and spiking neural networks (SNN) [3] as well as defect engineering for memristive characteristics improvement. In this work, we propose to combine RCS with SNN and create a bio-inspired neuromorphic system based on two types of organic memristors with specifically designed structures and advanced characteristics. The RCS consists of two main parts: the reservoir and the readout [2]. The reservoir layer extracts some representative features from the input data due to its internal nonlinear dynamics. The readout layer then uses these features to classify the input data. Typically, a conventional fully connected neural network is used as a readout layer in the RCS. The training process occurs only in the readout layer, while a reservoir is not trainable. This decrease in trainable parameters considerably reduces the memristive stochasticity impact on the training process. The use of different types of memristors for the RCS is essential. The reservoir layer should consist of memristors with short-term memory, i.e., volatile memristors. This way, memristors can process each input sample individually. Volatile polyaniline-based memristors were chosen for this layer implementation. They can operate within a biologically plausible time range, which is essential as we aim to mimic biological systems [4]. In contrast, the reservoir layer should consist of memristors with long-term memory, i.e., non-volatile memristors, because the readout layer should preserve the trained synaptic weights. Non-volatile parylene memristors with incorporated MoO3 nanoparticles were chosen for the readout layer. The reservoir computing system adopts some essential principles of brain function, as both short- and long-term memory are significant in biological systems. However, traditional neural networks are commonly used as a readout layer in the RCSs [2]. Their training requires global weight updates, making them vulnerable to memristive stochasticity. In contrast, the SNNs allow local training, e.g., using bio-inspired learning rules, which makes them more effective and robust [3]. Consequently, we presume that a fully organic RCS with an SNN readout layer is a promising hardware memristive architecture. The work consists of two parts: hardware and software. First, the polyaniline- and parylene-based memristive devices were fabricated and tested. Hardware polyaniline reservoir demonstrated an ability to extract characteristic features from the input data. Nanocomposite parylene memristors were suitable for the role of synapses in the readout layer due to the unique combination of high switching speed, high stability, low power consumption and the possibility of crossbar implementation. Next, the traditional and spiking readout layers were compared in simulation. It was shown that the SNN readout layer is more adaptive and sustainable to noise in image classification tasks as well as memristive stochasticity [5].

Reconfigurable Cascaded Thermal Neuristors for Neuromorphic Computing.

While the complementary metal-oxide semiconductor (CMOS) technology is the mainstream for the hardware implementation of neural networks, an alternative route is explored based on a new class of spiking oscillators called "thermal neuristors", which operate and interact solely via thermal processes. Utilizing the insulator-to-metal transition (IMT) in vanadium dioxide, a wide variety of reconfigurable electrical dynamics mirroring biological neurons is demonstrated. Notably, inhibitory functionality is achieved just in a single oxide device, and cascaded information flow is realized exclusively through thermal interactions. To elucidate the underlying mechanisms of the neuristors, a detailed theoretical model is developed, which accurately reflects the experimental results. This study establishes the foundation for scalable and energy-efficient thermal neural networks, fostering progress in brain-inspired computing.