Resistive Crossbar Research Articles

Parasitic-Aware Modeling and Neural Network Training Scheme for Energy-Efficient Processing-in-Memory With Resistive Crossbar Array

This paper presents a parasitic-aware modeling method for fast and accurate simulation of Processing-in-Memory (PIM) neural network (NN) implemented in resistive memristor crossbar array. This work proposed an efficient and accurate crossbar line resistance estimation model named <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\gamma $ </tex-math></inline-formula> -compact model, and the associated NN training scheme that takes the impact of line resistance into consideration with time complexity of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O(mn)</i> for a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{m}\times \text{n}$ </tex-math></inline-formula> resistive crossbar. The impact of the crossbar array parasitics to the vector-matrix multiplication (VMM) computation accuracy and multi-layer PIM NN inference accuracy are analyzed in detail. The advantage of the proposed model is demonstrated by a multilayer perceptron (MLP) of size <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$784\times 128\times 10$ </tex-math></inline-formula> implemented with resistive random access memory (RRAM) crossbar arrays for MNIST hand-written digits classification. The proposed method reduced the VMM computation error by 186 and 17 times compared to uncompensated method and the state-of-art compensation method, respectively. Maximum 98.1% inference accuracy is achieved with only 0.17% degradation compared to the ideal model.

A Fully Configurable PUF Using Dynamic Variations of Resistive Crossbar Arrays

A resistive random access memory (RRAM) as an emerging nanoelectronic device, is widely used for memory and physical unclonable function (PUF) applications. The compatibility of RRAM PUFs with memory architectures can be exploited to reduce the hardware overhead. Therefore, an intrinsic PUF using dynamic variations of resistive crossbar arrays is presented in this paper. Based on an improved sense amplifier (SA), the proposed intrinsic RRAM PUF can be fully configured as a memory cell or a PUF cell, leading to a minimal design overhead. Using the device-to-device (D2D) variation of an RRAM, a significant number of challenge-response pairs (CRPs) is generated with a flexible configuration of the resistive crossbar arrays. The proposed RRAM PUF can be refreshed to a new instance relying on the cycle-to-cycle (C2C) variation of an RRAM. An efficient challenge generation method is presented to improve the security of the proposed RRAM PUF. To verify the performance of the proposed RRAM PUF, 20 instances are simulated and implemented using a compact Spice model and a UMC 65 nm CMOS process technology, respectively. The simulation results show that the proposed RRAM PUF exhibits good performances with a high uniqueness, reliability, and reconfigurability. The randomness of the PUF is evaluated by the National Institute of Standards and Technology (NIST) and autocorrelation function (ACF) tests. Moreover, the experimental results show that the proposed RRAM PUF achieves a good resistance against machine learning (ML) attacks.

TxSim: Modeling Training of Deep Neural Networks on Resistive Crossbar Systems

Deep neural networks (DNNs) have gained tremendous popularity in recent years due to their ability to achieve superhuman accuracy in a wide variety of machine learning tasks. However, the compute and memory requirements of DNNs have grown rapidly, creating a need for energy-efficient hardware. Resistive crossbars have attracted significant interest in the design of the next generation of DNN accelerators due to their ability to natively execute massively parallel vector-matrix multiplications within dense memory arrays. However, crossbar-based computations face a major challenge due to device and circuit-level nonidealities, which manifest as errors in the vector-matrix multiplications and eventually degrade DNN accuracy. To address this challenge, there is a need for tools that can model the functional impact of nonidealities on DNN training and inference. Existing efforts toward this goal are either limited to inference or are too slow to be used for large-scale DNN training. We propose TxSim, a fast and customizable modeling framework to functionally evaluate DNN training on crossbar-based hardware considering the impact of nonidealities. The key features of TxSim that differentiate it from prior efforts are: 1) it comprehensively models nonidealities during all training operations (forward propagation, backward propagation, and weight update) and 2) it achieves computational efficiency by mapping crossbar evaluations to well-optimized Basic Linear Algebra Subprograms (BLAS) routines and incorporates speedup techniques to further reduce simulation time with minimal impact on accuracy. TxSim achieves 6×- 108× improvement in simulation speed over prior works, and thereby makes it feasible to evaluate the training of large-scale DNNs on crossbars. Our experiments using TxSim reveal that the accuracy degradation in DNN training due to nonidealities can be substantial (3%-36.4%) for large-scale DNNs and data sets, underscoring the need for further research in mitigation techniques. We also analyze the impact of various device and circuit-level parameters and the associated nonidealities to provide key insights that can guide the design of crossbar-based DNN training accelerators.

Power-efficient Spike Sorting Scheme Using Analog Spiking Neural Network Classifier

The method to map the neural signals to the neuron from which it originates is spike sorting. A low-power spike sorting system is presented for a neural implant device. The spike sorter constitutes a two-step trainer module that is shared by the signal acquisition channel associated with multiple electrodes. A low-power Spiking Neural Network (SNN) module is responsible for assigning the spike class. The two-step shared supervised on-chip training module is presented for improved training accuracy for the SNN. Post implant, the relatively power-hungry training module can be activated conditionally based on a statistics-driven retraining algorithm that allows on the fly training and adaptation. A low-power analog implementation for the SNN classifier is proposed based on resistive crossbar memory exploiting its approximate computing nature. Owing to the direct mapping of SNN functionality using physical characteristics of devices, the analog mode implementation can achieve ∼21 × lower power than its fully digital counterpart. We also incorporate the effect of device variation in the training process to suppress the impact of inevitable inaccuracies in such resistive crossbar devices on the classification accuracy. A variation-aware, digitally calibrated analog front-end is also presented, which consumes less than ∼50 nW power and interfaces with the digital training module as well as the analog SNN spike sorting module. Hence, the proposed scheme is a low-power, variation-tolerant, adaptive, digitally trained, all-analog spike sorter device, applicable to implantable and wearable multichannel brain-machine interfaces.

RxNN: A Framework for Evaluating Deep Neural Networks on Resistive Crossbars

Resistive crossbars designed with nonvolatile memory devices have emerged as promising building blocks for deep neural network (DNN) hardware, due to their ability to compactly and efficiently realize vector-matrix multiplication (VMM), the dominant computational kernel in DNNs. However, a key challenge with resistive crossbars is that they suffer from a range of device and circuit level nonidealities, such as driver resistance, sensing resistance, sneak paths, interconnect parasitics, nonlinearities in the peripheral circuits, stochastic write operations, and process variations. These nonidealities can lead to errors in VMMs, eventually degrading the DNN's accuracy. It is therefore critical to study the impact of crossbar nonidealities on the accuracy of large-scale DNNs (with millions of neurons and billions of synaptic connections). However, this is challenging because the existing device and circuit models are too slow to use in application-level evaluations. We present RxNN, a fast and accurate simulation framework to evaluate large-scale DNNs on resistive crossbar systems. RxNN splits and maps the computations involved in each DNN layer into crossbar operations, and evaluates them using a fast crossbar model (FCM) that accurately captures the errors arising due to crossbar nonidealities while being four-to-five orders of magnitude faster than circuit simulation. FCM models a crossbar-based VMM operation using three stages-nonlinear models for the input and output peripheral circuits (digital-to-analog and analog-to-digital converters), and an equivalent nonideal conductance matrix for the core crossbar array. We implement RxNN by extending the Caffe machine learning framework and use it to evaluate a suite of six large-scale DNNs developed for the ImageNet Challenge (ILSVRC). Our experiments reveal that resistive crossbar nonidealities can lead to significant accuracy degradations (9.6%-32%) for these large-scale DNNs. To the best of our knowledge, this article is the first quantitative evaluation of the accuracy of large-scale DNNs on resistive crossbar-based hardware. We also demonstrate that RxNN enables fast model-in-the-loop retraining of DNNs to partially mitigate the accuracy degradation.

Mitigate Parasitic Resistance in Resistive Crossbar-based Convolutional Neural Networks

Traditional computing hardware often encounters on-chip memory bottleneck on large-scale Convolution Neural Networks (CNN) applications. With its unique in-memory computing feature, resistive crossbar-based computing attracts researchers’ attention as a promising solution to the memory bottleneck issue in von Neumann architectures. However, the parasitic resistances in crossbar deviate its behavior from the ideal weighted summation operation. In large-scale implementations, the impact of parasitic resistances must be carefully considered and mitigated to ensure circuits’ functionality. In this work, we implemented and simulated CNNs on resistive crossbar circuits with consideration of parasitic resistances. Moreover, we carried out a new mapping scheme for high utilization of crossbar arrays on convolution, and a mitigation algorithm to mitigate parasitic resistances in CNN applications. The mitigation algorithm considers parasitic resistances as well as data/kernel patterns of each layer to minimize the computing error in crossbar-based convolutions of CNNs. We demonstrated the proposed methods with implementations of a 4-layer CNN on MNIST, and residual neural network (ResNet) (20, 32, and 56) on CIFAR-10. Simulation results show the proposed methods well mitigate the parasitic resistances in crossbars. With our methods, modern CNNs on crossbars can preserve ideal (software) level classification accuracy with 6-bit ADCs and DACs implementation.

IR-QNN Framework: An IR Drop-Aware Offline Training of Quantized Crossbar Arrays

Resistive Crossbar Arrays present an elegant implementation solution for Deep Neural Networks acceleration. The Matrix-Vector Multiplication, which is the corner-stone of DNNs, is carried out in $O(1)$ compared to $O(N^{2})$ steps for digital realizations of $O(log_{2}(N))$ steps for in-memory associative processors. However, the IR drop problem, caused by the inevitable interconnect wire resistance in RCAs remains a daunting challenge. In this article, we propose a fast and efficient training and validation framework to incorporate the wire resistance in Quantized DNNs, without the need for computationally extensive SPICE simulations during the training process. A fabricated four-bit Au/Al2O3/HfO2/TiN device is modelled and used within the framework with two-mapping schemes to realize the quantized weights. Efficient system-level IR-drop estimation methods are used to accelerate training. SPICE validation results show the effectiveness of the proposed method to capture the IR drop problem achieving the baseline accuracy with a 2% and 4% drop in the worst-case scenario for MNIST dataset on multilayer perceptron network and CIFAR 10 dataset on modified VGG and AlexNet networks, respectively. Other nonidealities, such as stuck-at fault defects, variability, and aging, are studied. Finally, the design considerations of the neuronal and the driver circuits are discussed.

Training Large-scale Artificial Neural Networks on Simulated Resistive Crossbar Arrays

Accelerating training of artificial neural networks (ANN) with analog resistive crossbar arrays is a promising idea. While the concept has been verified on very small ANNs and toy data sets (such as MNIST), more realistically sized ANNs and datasets have not yet been tackled. However, it is to be expected that device materials and hardware design constraints, such as noisy computations, finite number of resistive states of the device materials, saturating weight and activation ranges, and limited precision of analog-to-digital converters, will cause significant challenges to the successful training of state-of-the-art ANNs. By using analog hardware aware ANN training simulations, we here explore a number of simple algorithmic compensatory measures to cope with analog noise and limited weight and output ranges and resolutions, that dramatically improve the simulated training performances on RPU arrays on intermediately to large-scale ANNs.

CxDNN

Resistive crossbars have shown strong potential as the building blocks of future neural fabrics, due to their ability to natively execute vector-matrix multiplication (the dominant computational kernel in DNNs). However, a key challenge that arises in resistive crossbars is that non-idealities in the synaptic devices, interconnects, and peripheral circuits of resistive crossbars lead to errors in the computations performed. When large-scale DNNs are executed on resistive crossbar systems, these errors compound and result in unacceptable degradation in application-level accuracy. We propose CxDNN, a hardware-software methodology that enables the realization of large-scale DNNs on crossbar systems by compensating for errors due to non-idealities, greatly mitigating the degradation in accuracy. CxDNN is composed of (i) an optimized mapping technique to convert floating-point weights and activations to crossbar conductances and input voltages, (ii) a fast one-time re-training method to recover accuracy loss due to this conversion, and (iii) low-overhead compensation hardware to mitigate dynamic and hardware-instance-specific errors. Unlike previous efforts that are limited to small networks and require the training and deployment of hardware-instance-specific models, CxDNN presents a scalable compensation methodology that can address large DNNs (e.g., ResNet-50 on ImageNet) and maintains the train-once-deploy-anywhere tenet of current DNN application. We evaluated CxDNN on six top DNNs on the ImageNet dataset with 0.5--13.8 million neurons and 0.5--15.5 billion connections. CxDNN achieves 16.9%--49% improvement in the top-1 classification accuracy, effectively mitigating a key challenge to the use of resistive crossbar--based neural fabrics.

Design and characterization of superconducting nanowire-based processors for acceleration of deep neural network training

Training of deep neural networks (DNNs) is a computationally intensive task and requires massive volumes of data transfer. Performing these operations with the conventional von Neumann architectures creates unmanageable time and power costs. Recent studies have shown that mixed-signal designs involving resistive crossbar architectures are capable of achieving acceleration factors as high as 30 000 × over the state of the art digital processors. These approaches involve utilization of non-volatile memory elements as local processors. However, no technology has been developed to-date that can satisfy the strict device requirements for the unit cell. This paper presents the superconducting nanowire-based processing element as a crosspoint device. The unit cell has many programmable non-volatile states that can be used to perform analog multiplication. Importantly, these states are intrinsically discrete due to quantization of flux, which provides symmetric switching characteristics. Operation of these devices in a crossbar is described and verified with electro-thermal circuit simulations. Finally, validation of the concept in an actual DNN training task is shown using an emulator.

A Power and Area Efficient CMOS Stochastic Neuron for Neural Networks Employing Resistive Crossbar Array.

A power and area efficient CMOS stochastic neuron for resistive computing device-based neural networks is presented. The stochastic neuron performs both quantization and activation function simultaneously by using a single dynamic comparator and allows power-hungry analog to digital and digital to analog converters to be removed at the cost of the increased computation time. A network learning method utilizing a noisy sigmoid function is also presented to minimize the computation time with little accuracy degradation. A prototype neuron chip fabricated in 0.18μm CMOS process successfully demonstrates the neuron's performance and the learning method is verified through network simulations.

Mask Technique for Fast and Efficient Training of Binary Resistive Crossbar Arrays

Resistive crossbar arrays, despite their advantages such as parallel processing, low power, and high density, suffer from the sneak path problem created by IR voltage drops. The crossbar non-idealities considerably degrade the accuracy of matrix multiplication operation, which is the cornerstone of hardware accelerated neural networks. In this paper, we propose a novel technique to capture the sneak path problem to enable off-chip learning and weight transfer. The proposed technique can be easily integrated into any neural network training framework. Performance results show a significant improvement after retraining the network with the proposed mask technique. Two mask solutions have been proposed and studied to capture the sneak path problem in resistive crossbar arrays. Both mask solutions were successfully able to achieve classification accuracy close to the baseline accuracy. The tradeoffs between the two solutions are discussed and compared in terms of accuracy, power, and area.

Analog Spike-Timing-Dependent Resistive Crossbar Design for Brain Inspired Computing

With the rapid advancement in the complementary metal-oxide-semiconductor (CMOS) technology, the performance of neuromorphic computing systems that employed the emerging devices has entered a new era of high accuracy and energy efficient operations. As one of the most fundamental components in computing systems, encoding scheme characterizes the relationship between the stimulus and the individual or ensemble neuronal responses. Although rate encoding is easier to implement and with high error tolerance, temporal encoding offers high data density and energy efficiency. In this paper, an inter-spike interval (ISI)-based resistive crossbar neuromorphic design, built with the standard CMOS technology, is proposed. Our proposed decoder exhibits not only computation accuracy but also robustness. Another worth-mentioning achievement is the data compressor in our work, which is based on the spike temporal encoding scheme. In this paper, design and performance analysis for proposed ISI-based resistive crossbar are elaborated; as well as the application of our proposed design. 24 patterns are adopted to encode the sensory information which are successfully represented by seven inter-spike intervals leading to significantly high compression rate. To evaluate the performance, a test bench of video frames consisting one person rotating her head from 0° to 75° with an increment of 15° has been employed. The results showed that the ISI code has better performance in both recognition rate and converging speed.

Modeling Size Limitations of Resistive Crossbar Array With Cell Selectors

Due to recent developments in emerging memory technologies, resistive crossbar arrays have gained increasing importance. The size of the crossbar arrays is, however, limited due to challenges brought by the interconnect resistance, sneak path currents, and the physical area of the peripheral circuitry. In this paper, three figures of merit that characterize the limitations of resistive crossbar arrays with selectors are described, such as the driver resistance, voltage degradation across the cell, and read margin. The models, exhibiting good agreement with SPICE, are compared with different biasing schemes during both write and read operations. These models are also used to predict the device requirements of resistive crossbar arrays with selectors and to project parameter values, such as the nonlinearity factor, on-state resistance, and tolerable interconnect resistance per cell for large-scale crossbar arrays.

Associative Memristive Memory for Approximate Computing in GPUs

Using associative memories to enable computing-with-memory is a promising approach to improve energy efficiency. Associative memories can be tightly coupled with processing elements to restore and later recall function responses for a subset of input values. This approach avoids the actual function execution on the processing element to save on energy. The challenge, however, is to reduce the energy consumption of associative memory modules themselves. Here we address the challenge of designing ultra-low-power associative memories. We use memristive parts for memory implementation and demonstrate the energy saving potential of integrating associative memristive memory (AMM) into graphics processing units (GPUs). To reduce the energy consumption of AMM modules, we leverage approximate computing which benefits from application-level tolerance to errors: We employ voltage overscaling on AMM modules which deliberately relaxes its searching criteria to approximately match stored patterns within a 2 bit Hamming distance of the search pattern. This introduces some errors to the computation that are tolerable for target applications. We further reduce the energy consumption by employing purely resistive crossbar architectures for AMM modules. To evaluate the proposed architecture, we integrate AMM modules with floating point units in an AMD Southern Islands GPU and run four image processing kernels on an AMM-integrated GPU. Our experimental results show that employing AMM modules reduces energy consumption of running these kernels by 23%–45%, compared to a baseline GPU without AMM. The image processing kernels tolerate errors resulting from approximate search operations, maintaining an acceptable image quality, i.e., a PSNR above 30 dB.

Resistive GP-SIMD Processing-In-Memory

GP-SIMD, a novel hybrid general-purpose SIMD architecture, addresses the challenge of data synchronization by in-memory computing, through combining data storage and massive parallel processing. In this article, we explore a resistive implementation of the GP-SIMD architecture. In resistive GP-SIMD, a novel resistive row and column addressable 4F 2 crossbar is utilized, replacing the modified CMOS 190F 2 SRAM storage previously proposed for GP-SIMD architecture. The use of the resistive crossbar allows scaling the GP-SIMD from few millions to few hundred millions of processing units on a single silicon die. The performance, power consumption and power efficiency of a resistive GP-SIMD are compared with the CMOS version. We find that PiM architectures and, specifically, GP-SIMD benefit more than other many-core architectures from using resistive memory. A framework for in-place arithmetic operation on a single multivalued resistive cell is explored, demonstrating a potential to become a building block for next-generation PiM architectures.

Hierarchical Temporal Memory Based on Spin-Neurons and Resistive Memory for Energy-Efficient Brain-Inspired Computing.

Hierarchical temporal memory (HTM) tries to mimic the computing in cerebral neocortex. It identifies spatial and temporal patterns in the input for making inferences. This may require a large number of computationally expensive tasks, such as dot product evaluations. Nanodevices that can provide direct mapping for such primitives are of great interest. In this paper, we propose that the computing blocks for HTM can be mapped using low-voltage, magnetometallic spin-neurons combined with an emerging resistive crossbar network, which involves a comprehensive design at algorithm, architecture, circuit, and device levels. Simulation results show the possibility of more than 200× lower energy as compared with a 45-nm CMOS ASIC design.

A &lt;inline-formula&gt; &lt;tex-math notation="LaTeX"&gt;$\mu $ &lt;/tex-math&gt;&lt;/inline-formula&gt;-Controller-Based System for Interfacing Selectorless RRAM Crossbar Arrays

Selectorless crossbar arrays of resistive random-access memory (RRAM), also known as memristors, conduct large sneak currents during operation, which can significantly corrupt the accuracy of cross-point analog resistance ( $M_{t}$ ) measurements. In order to mitigate this issue, we have designed, built, and tested a memristor characterization and testing (mCAT) instrument that forces redistribution of sneak currents within the crossbar array, dramatically increasing $M_{t}$ measurement accuracy. We calibrated the mCAT using a custom-made $32 \times 32$ discrete resistive crossbar array, and subsequently demonstrated its functionality on solid-state TiO2- x RRAM arrays, on wafer and packaged, of the same size. Our platform can measure standalone $M_{t}$ in the range of 1 $\text{k}\Omega $ to 1 $\text{M}\Omega $ with <1% error. For our custom resistive crossbar, 90% of devices of the same resistance range were measured with <10% error. The platform’s limitations have been quantified using large-scale nonideal crossbar simulations.

Nano-Crossbar Memories Comprising Parallel/Serial Complementary Memristive Switches

This work explores anti-serial (anti-parallel) memristive switches—ASMs (APMs)—as potential cross-point elements in nano-crossbar resistive random access memory arrays. The memory operation principles for both device combinations are shown in detail. The effectiveness of these memristive structures to the solution of the parasitic conducting (current sneak paths) problem is presented via an analytical approach which is based on the basic setup of resistive crossbar memories. Simulation results of crossbars of up to 4,096 elements, arranged in quadratic configurations, are conducted. The provided results supplement this comprehensive analysis of APMs and ASMs, outlining their overall performance characteristics and commenting on their applicability to the practical realization of large crossbar memory systems. Finally, a special array topology is applied to an ASM-based crossbar memory. Its performance is compared to the performance of the pure ASM-based memory. The conducted simulations reveal significantly improved read-out voltage margins which further contribute to addressing the parasitic current paths which prevent the reliable operation of memristive crossbar circuit topologies.

Influence of the Interconnection Line Resistance and Performance of a Resistive Cross Bar Array Memory

The effects of the external load resistance on the resistive switching (RS) behavior of RS cells were examined using model calculations and experiments. With increasing load resistance, the reset voltage increased more rapidly than the set voltage, which eventually resulted in RS failure. For the experiment, various electrode materials were examined to achieve both a stable RS behavior of a film and a cross bar array (CBA) with sufficiently low line resistance. The effect of the line resistance on the RS behavior was studied from the electrical characterization of the fabricated cross-bar structure. A high line resistance causes an undesirable high operation voltage and the failure of stable RS in extreme cases. Overall, a sufficiently low interconnect line resistance is essential for achieving stable operation and rapid RS in a CBA structure.