Artificial Neural Networks Hardware Research Articles

A linear compensation method for inference accuracy improvement of memristive in-memory computing.

Memristive computing system (MCS), with the feature of in-memory computing capability, for artificial neural networks (ANNs) deployment showing low power and massive parallelism, is a promising alternative for traditional Von-Neumann architecture computing system. However, because of the various non-idealities of both peripheral circuits and memristor array, the performance of the practical MCS tends to be significantly reduced. In this work, a linear compensation method (LCM) is proposed for the performance improvement of MCS under the effect of non-idealities. By considering the effects of various non-ideal states in the MCS as a whole, the output error of the MCS under different conditions is investigated. Then, a mathematic model for the output error is established based on the experimental data. Furthermore, the MCS is researched at the physical circuit level as well, in order to analyze the specific way in which the non-idealities affect the output current. Finally, based on the established mathematical model, the LCM output current is compensated in real time to improve the system performance. the effectiveness of LCM is verified and showing outstanding performance in the ResNet-34 network architecture, which is easily affected by the non-idealities in hardware. The proposed LCM can be naturally integrated into the operation processes of MCS, paving the way for optimizing the deployment on generic ANN hardware based on the memristor.&#xD.

An Artificial Neural Network Based on Oxide Synaptic Transistor for Accurate and Robust Image Recognition.

Synaptic transistors with low-temperature, solution-processed dielectric films have demonstrated programmable conductance, and therefore potential applications in hardware artificial neural networks for recognizing noisy images. Here, we engineered AlOx/InOx synaptic transistors via a solution process to instantiate neural networks. The transistors show long-term potentiation under appropriate gate voltage pulses. The artificial neural network, consisting of one input layer and one output layer, was constructed using 9 × 3 synaptic transistors. By programming the calculated weight, the hardware network can recognize 3 × 3 pixel images of characters z, v and n with a high accuracy of 85%, even with 40% noise. This work demonstrates that metal-oxide transistors, which exhibit significant long-term potentiation of conductance, can be used for the accurate recognition of noisy images.

Graphene oxide-DNA/graphene oxide-PDDA sandwiched membranes with neuromorphic function.

The behavior of polyelectrolytes in confined spaces has direct relevance to the protein mediated ion transport in living organisms. In this paper, we govern lithium chloride transport by the interface provided by polyelectrolytes, polycation, poly(diallyldimethylammonium chloride) (PDDA) and, polyanion, double stranded deoxyribonucleic acid (dsDNA), in confined graphene oxide (GO) membranes. Polyelectrolyte-GO interfaces demonstrate neuromorphic functions that were successfully applied with nanochannel ion interactions contributed, resulting in ion memory effects. Excitatory and inhibitory post-synaptic currents were tuned continuously as the number of pulses applied increased accordingly, increasing decay times. Furthermore, we demonstrated the short-term memory of a trained vs untrained device in computation. On account of its simple and safe production along with its robustness and stability, we anticipate our device to be a low dimensional building block for arrays to embed artificial neural networks in hardware for neuromorphic computing. Additionally, incorporating such devices with sensing and actuating parts for a complete feedback loop produces robotics with its own ability to learn by modifying actuation based on sensing data.

Photonic Neural Networks Based on Integrated Silicon Microresonators

Recent progress in artificial intelligence (AI) has boosted the computational possibilities in fields in which standard computers are not able to perform adequately. The AI paradigm is to emulate human intelligence and therefore breaks the familiar architecture on which digital computers are based. In particular, neuromorphic computing, artificial neural networks (ANNs), and deep learning models mimic how the brain computes. There are many applications for large networks of interconnected neurons whose synapses are individually strengthened or weakened during the learning phase. In this respect, photonics is a suitable platform for implementing ANN hardware owing to its speed, low power dissipation, and multi-wavelength opportunities. One photonic device that could serve as an optical neuron is the optical microring resonator. Indeed, microring resonators exhibit a nonlinear response and the capability for optical energy storage, which can be used to implement fading memory. In addition, their characteristic resonant behavior makes them extremely sensitive to input wavelengths, which promotes wavelength division multiplexing (WDM) applications and enables their use as WDM-based synapses (weight banks) in the linear regime. Remarkably, using silicon photonics, photonic integrated circuits can be fabricated in volume and with integrated electronics onboard. For these reasons, here, we describe the physics of silicon microring resonators and arrays of microring resonators for application in neuromorphic computing. We describe different types of ANNs, from feedforward networks to photonic extreme learning machines, and reservoir computing. In addition, we discuss hybrid systems in which silicon microresonators are coupled with other active materials. This review introduces the basics and discusses the most recent developments in the field.

Impact of High-Level-Synthesis on Reliability of Artificial Neural Network Hardware Accelerators

Impact of High-Level-Synthesis on Reliability of Artificial Neural Network Hardware Accelerators

Generative complex networks within a dynamic memristor with intrinsic variability

Artificial neural networks (ANNs) have gained considerable momentum in the past decade. Although at first the main task of the ANN paradigm was to tune the connection weights in fixed-architecture networks, there has recently been growing interest in evolving network architectures toward the goal of creating artificial general intelligence. Lagging behind this trend, current ANN hardware struggles for a balance between flexibility and efficiency but cannot achieve both. Here, we report on a novel approach for the on-demand generation of complex networks within a single memristor where multiple virtual nodes are created by time multiplexing and the non-trivial topological features, such as small-worldness, are generated by exploiting device dynamics with intrinsic cycle-to-cycle variability. When used for reservoir computing, memristive complex networks can achieve a noticeable increase in memory capacity a and respectable performance boost compared to conventional reservoirs trivially implemented as fully connected networks. This work expands the functionality of memristors for ANN computing.

An efficient design methodology to speed up the FPGA implementation of artificial neural networks

In this paper, we propose and formulate a C++ based training methodology for speeding up the implementation of an Artificial Neural Network (ANN) in a Field Programmable Gate Array (FPGA). The proposed ANN implementation methodology uses a custom C++ based program referred to as Neural Network Design Parameter Extraction (NNDPE) program developed using an open source library namely Fast Artificial Neural Network (FANN). The proposed C++ based NNDPE program reduces the time required to train and extract the design parameters of the ANN, such as the number of layers, the number of neurons in each layer, and the weights. The extracted ANN design parameters, custom hardware arithmetic units, and function-approximate activation functions are used to implement the ANN hardware architecture of a linear function on the Virtex-7 FPGA platform. The Vivado 2018.3 tool is used to simulate, synthesize, and implement the ANN-based linear function. It was concluded from the simulation that the ANN hardware implementation has high precision due to the usage of floating-point arithmetic operations compared to the ANN which uses fixed-point arithmetic operations. The ANN-implemented on the Virtex-7 FPGA operates at 150.76 MHz, which is approximately 11 to 18 times faster than the software implementation running on various CPU cores with operating frequencies ranging over 2.2 GHz to 4.70 GHz.

Are SNNs Really More Energy-Efficient Than ANNs? an In-Depth Hardware-Aware Study

Spiking Neural Networks (SNNs) hold the promise of lower energy consumption in embedded hardware due to their spike-based computations compared to traditional Artificial Neural Networks (ANNs). The relative energy efficiency of this emerging technology compared to traditional digital hardware has not been fully explored. Many studies do not consider memory accesses, which account for an important fraction of the energy consumption, use naive ANN hardware implementations, or lack generality. In this paper, we compare the relative energy efficiency of classical digital implementations of ANNs with novel event-based SNN implementations based on variants of the Integrate and Fire (IF) model. We provide a theoretical upper bound on the relative energy efficiency of ANNs, by computing the maximum possible benefit from ANN data reuse and sparsity. We also use the Eyeriss ANN accelerator as a case study. We show that the simpler IF model is more energy-efficient than the Leaky IF and temporal continuous synapse models. Moreover, SNNs with the IF model can compete with efficient ANN implementations when there is a very high spike sparsity, i.e. between 0.15 and 1.38 spikes per synapse per inference, depending on the ANN implementation. Our analysis shows that hybrid ANN-SNN architectures, leveraging a SNN event-based approach in layers with high sparsity and ANN parallel processing for the others, are a promising new path for further energy savings.

Design of artificial neural network hardware accelerator

We present a design of the scalable processor capable of providing an artificial neural network (ANN) functionality and in-house developed tools for automatic conversion of an ANN model designed with the TensorFlow library into an HDL code. The hardware is described in SystemVerilog and the synthesized module of the processor can perform calculations of a neural network with the speed exceeding 100 MHz. Our in-house designed software tool for ANN conversion supports translation of an arbitrary multilayer perceptron neural network into a state machine module, which performs necessary calculations. It is also dynamically reconfigurable so that the ANN operating on the hardware can be changed after it is deployed as an ASIC. The project aims the in-pixel implementation towards an X-ray photon energy estimation. The energy estimation shall be delivered with accuracy that exceeds the accuracy of an ADC converter that feeds the ANN with data.

Benchmarking energy consumption and latency for neuromorphic computing in condensed matter and particle physics

The massive use of artificial neural networks (ANNs), increasingly popular in many areas of scientific computing, rapidly increases the energy consumption of modern high-performance computing systems. An appealing and possibly more sustainable alternative is provided by novel neuromorphic paradigms, which directly implement ANNs in hardware. However, little is known about the actual benefits of running ANNs on neuromorphic hardware for use cases in scientific computing. Here, we present a methodology for measuring the energy cost and compute time for inference tasks with ANNs on conventional hardware. In addition, we have designed an architecture for these tasks and estimate the same metrics based on a state-of-the-art analog in-memory computing (AIMC) platform, one of the key paradigms in neuromorphic computing. Both methodologies are compared for a use case in quantum many-body physics in two-dimensional condensed matter systems and for anomaly detection at 40 MHz rates at the Large Hadron Collider in particle physics. We find that AIMC can achieve up to one order of magnitude shorter computation times than conventional hardware at an energy cost that is up to three orders of magnitude smaller. This suggests great potential for faster and more sustainable scientific computing with neuromorphic hardware.

Flexible Active Crossbar Arrays Using Amorphous Oxide Semiconductor Technology toward Artificial Neural Networks Hardware

AbstractMemristor crossbar arrays can compose the efficient hardware for artificial intelligent applications. However, the requirements for a linear and symmetric synaptic weight update and low cycle‐to‐cycle (C2C) and device‐to‐device variability as well as the sneak‐path current issue have been delaying its further development. This study reports on a thin‐film amorphous oxide‐based 4×4 1‐transistor 1‐memristor (1T1M) crossbar. The a‐IGZO crossbar is built on a flexible polyimide substrate, enabling IoT and wearable applications. In the novel framework, the thin‐film transistor and memristor are fabricated at the same level, with the same processing steps and sharing the same materials for all layers. The 1T1M cells show linear and symmetrical plasticity characteristic with low C2C variability. The memristor performs like an analog dot product engine and vector–matrix multiplications in the 4×4 crossbars is demonstrated experimentally, in which the sneak‐path current issue is successfully suppressed, resulting in a proof‐of‐concept for a cost‐effective, flexible artificial neural networks hardware.

Multilevel switching memristor by compliance current adjustment for off-chip training of neuromorphic system

Multilevel operation is one of the most essential properties for synaptic devices to realize hardware artificial neural networks. Compliance current (Icc) adjustment is a multilevel programming method that can be utilized for a large-scale one-transistor and one-resistor (1T1R) array. It protects the devices from permanent breakdown by regulating abrupt switching. However, according to the reported literature so far, the number of conductance states in the Icc control method is insufficient to implement off-chip-trained neuromorphic systems. Therefore, we experimentally explore the feasibility of a larger number of conductance states using the Icc control method. We fabricated an Al2O3/TiOx-based resistive switching memory array, observed the conductance change while increasing Icc during set operations, and 64-level conductance states were statistically demonstrated. Furthermore, we verified that the 64-level states showed recognition performance close to that of a software-based neural network through off-chip learning of the convolutional neural network (CNN) structure. The fabricated synaptic device array with the Icc-control programming method is expected to contribute to the development of hardware neural network by reducing the information loss in the transfer process.

Reversible training of waveguide-based AND/OR gates for optically driven artificial neural networks using photochromic molecules

Artificial neural networks (ANNs) are inspired by the biological nervous system. The high performance of such ANNs is achieved through the dynamic change of the synaptic weights by applying self-optimizing learning algorithms. Despite the simple operations for each single element in an ANN, a network with a huge number of simulated elements consumes lots of computing capacity using von Neumann computer architectures. To overcome this issue, neuromorphic devices facilitate the design of hardware ANNs that emulate the synaptic functions. Here we demonstrate the viability of such an approach using photonic waveguides in combination with a photochromic diarylethene (DAE) molecule. By positioning and irradiating DAE molecules on single waveguides, we modulate the intensity and thereby emulate the plasticity of the synaptic weights. To run the photonic device as an ANN we firstly characterize the modulation range and encode a learning procedure accordingly. As the proof of concept, we operate a y-shaped waveguide performing basic AND/OR logic gate functions, with the capability to switch between these two gate functions by using specific training sets.

An accurate and fair evaluation methodology for SNN-based inferencing with full-stack hardware design space explorations

Artificial Neural Networks (ANNs) achieve high accuracy in various cognitive tasks (i.e., inferences), but often fail to meet power and latency budgets due to intensive computational overheads. To address the challenge, Spiking Neural Networks (SNNs) have emerged as high-performance and power-efficient alternatives thanks to their theoretically efficient spike-driven computations. The spike-based computations have a high potential of achieving cost-effective inferencing with their low-precision data representations, simple neuron operations, and new parallelization opportunities. To determine which network (i.e., ANN or SNN) is suitable for which purposes, it is essential to accurately evaluate the cost-effectiveness of an SNN and compare it to the corresponding ANN. However, existing studies overestimate or underestimate the cost-effectiveness of SNNs over ANNs as they consider only the limited design points and compare SNNs against naive ANN hardware baselines.In this study, we propose a full-stack SNN evaluation methodology to accurately evaluate the cost-effectiveness of SNNs. Quantifying the potential of SNNs is highly challenging as the evaluations require full-stack knowledge on SNNs’ unique computational features and how each affects the mechanisms of the underlying hardware. For this purpose, we identify five representative SNN-specific design points that affect hardware performance and demonstrate the impact of each design point with the quantified experimental results. Next, we modify the existing ANN accelerator to support the identified SNN-specific design points and implement a cycle-accurate simulator to evaluate how each point affects the overall cost-effectiveness. As a case study, we evaluate SNNs converted from pretrained ANNs and compare them against the ANN counterparts using our simulator. Our study is the first work to accurately evaluate the cost-effectiveness of SNNs and make fair comparisons against ANNs. In addition, our methodology provides important guidelines for the next-generation SNN accelerator designs.

Power and area efficient stochastic artificial neural networks using spin–orbit torque-based true random number generator

Hardware implementations of Artificial Neural Networks (ANNs) using conventional binary arithmetic units are computationally expensive and energy-intensive together with large area footprints. Stochastic computing (SC) is an unconventional computing paradigm that operates on stochastic bit streams. It can offer low-power and area-efficient hardware implementations and has shown promising results when applied to ANN hardware circuits. SC relies on stochastic number generators (SNGs) to map input binary numbers to stochastic bit streams. The SNGs are conventionally implemented using random number generators (RNGs) and comparators. Linear feedback shifted registers (LFSRs) are typically used as the RNGs, which need far more area and power than the SC core, counteracting the latter's main advantages. To mitigate this problem, in this Letter, RNGs employing Spin–Orbit Torque (SOT)-induced stochastic switching of perpendicularly magnetized Ta/CoFeB/MgO nanodevices have been proposed. Furthermore, the SOT true random number generator (TRNG) is integrated with the simple CMOS stochastic computing circuits to perform a stochastic artificial neural network. To further optimize power and area efficiency, a fully parallel architecture and TRNG-sharing scheme are presented. The proposed stochastic ANN using the SOT-based TRNG obtains a negligible inference accuracy loss, compared with the binary version, and achieves 9× and 25× improvement in terms of area and power, respectively, compared with the ANN using LFSRs.

Hardware Implementation of Neuromorphic Computing Using Large‐Scale Memristor Crossbar Arrays

Brain‐inspired neuromorphic computing is a new paradigm that holds great potential to overcome the intrinsic energy and speed issues of traditional von Neumann based computing architecture. With the ability to perform vector‐matrix multiplications and flexible tunable conductance, the memristor crossbar array (CBA) structure is one of the most promising candidates to realize neural cognitive systems. The boom in the development of memristive synapses and neurons has propelled the developments of artificial neural networks (ANNs) to emulate the highly hierarchically organized network of human brain in the past decade. To achieve this, realizing large scale, high‐density memristive CBAs is a prerequisite to constructing complex ANNs. Herein, the stringent requirements in device performance and array parameters for hardware ANNs are analyzed, and the efforts in addressing the associated challenges are discussed. Recent progress on the experimental demonstration of neuromorphic computing systems (NCSs) is presented. Recommendations for further performance optimization at the device, circuit, and algorithm levels are proposed. This Report serves as a guide for the hardware implementation of NCS based on large‐scale CBAs.

Committee machines\u2014a universal method to deal with non-idealities in memristor-based neural networks

Artificial neural networks are notoriously power- and time-consuming when implemented on conventional von Neumann computing systems. Consequently, recent years have seen an emergence of research in machine learning hardware that strives to bring memory and computing closer together. A popular approach is to realise artificial neural networks in hardware by implementing their synaptic weights using memristive devices. However, various device- and system-level non-idealities usually prevent these physical implementations from achieving high inference accuracy. We suggest applying a well-known concept in computer science—committee machines—in the context of memristor-based neural networks. Using simulations and experimental data from three different types of memristive devices, we show that committee machines employing ensemble averaging can successfully increase inference accuracy in physically implemented neural networks that suffer from faulty devices, device-to-device variability, random telegraph noise and line resistance. Importantly, we demonstrate that the accuracy can be improved even without increasing the total number of memristors.

Temperature-resilient solid-state organic artificial synapses for neuromorphic computing.

Devices with tunable resistance are highly sought after for neuromorphic computing. Conventional resistive memories, however, suffer from nonlinear and asymmetric resistance tuning and excessive write noise, degrading artificial neural network (ANN) accelerator performance. Emerging electrochemical random-access memories (ECRAMs) display write linearity, which enables substantially faster ANN training by array programing in parallel. However, state-of-the-art ECRAMs have not yet demonstrated stable and efficient operation at temperatures required for packaged electronic devices (~90°C). Here, we show that (semi)conducting polymers combined with ion gel electrolyte films enable solid-state ECRAMs with stable and nearly temperature-independent operation up to 90°C. These ECRAMs show linear resistance tuning over a >2× dynamic range, 20-nanosecond switching, submicrosecond write-read cycling, low noise, and low-voltage (±1 volt) and low-energy (~80 femtojoules per write) operation combined with excellent endurance (>109 write-read operations at 90°C). Demonstration of these high-performance ECRAMs is a fundamental step toward their implementation in hardware ANNs.

An Artificial Neural Network Processor With a Custom Instruction Set Architecture for Embedded Applications

This article presents the design and implementation of an embedded programmable processor with a custom instruction set architecture for efficient realization of artificial neural networks (ANNs). The ANN processor architecture is scalable, supporting an arbitrary number of layers and number of artificial neurons (ANs) per layer. Moreover, the processor supports ANNs with arbitrary interconnect structures among ANs to realize both feed-forward and dynamic recurrent networks. The processor architecture is customizable in which the numerical representation of inputs, outputs, and signals among ANs can be parameterized to an arbitrary fixed-point format. An ASIC implementation of the designed programmable ANN processor for networks with up to 512 ANs and 262,000 interconnects is presented and is estimated to occupy 2.23 mm2 of silicon area and consume 1.25 mW of power from a 1.6 V supply while operating at 74 MHz in a standard 32-nm CMOS technology. In order to assess and compare the efficiency of the designed ANN processor, we have designed and implemented a dedicated reconfigurable hardware architecture for the direct realization of ANNs. Characteristics and implementation results of the designed programmable ANN processor and the dedicated ANN hardware on a Xilinx Artix-7 field-programmable gate array (FPGA) are presented and compared using two benchmarks, the MNIST benchmark using a feed-forward ANN and a movie review sentiment analysis benchmark using a recurrent neural network.

Embedded Hardware Artificial Neural Network Control for Global and Real-Time Imbalance Current Suppression of Parallel Connected IGBTs

A global and real-time control with embedded hardware artificial neural network (ANN) for imbalance current suppression of parallel connected insulated gate bipolar transistors (IGBTs) is first proposed in this paper. This method focuses on control strategy and control execution. The former one is realized by porting the ANN-based PID (ANN-PID) strategy in the control loop to yield the real-time and adaptive characteristics without IGBT quantity limitation. The latter one is realized by designing the IGBT gate quantity of charge regulator (GQR) to execute the command from ANN-PID controller. The evaluation of ANN-PID algorithm results 0.023% mean error in IGBT current control that reveals the feasibility of the proposed method. A full prototype with FPGA-based hardware accelerator for ANN-PID computing, including the designed GQR circuit, has been built for realization and qualification in a buck converter with parallel connected IGBTs. The experimental results show that the performance of the proposed method in imbalance current suppression is improved about 3.5–5.5 times as the load increase from low to high with the advantage of immunity to load change and the current imbalance can be suppressed within 4%.