Abstract
Hardware architectures composed of resistive cross-point device arrays can provide significant power and speed benefits for deep neural network training workloads using stochastic gradient descent (SGD) and backpropagation (BP) algorithm. The training accuracy on this imminent analog hardware, however, strongly depends on the switching characteristics of the cross-point elements. One of the key requirements is that these resistive devices must change conductance in a symmetrical fashion when subjected to positive or negative pulse stimuli. Here, we present a new training algorithm, so-called the “Tiki-Taka” algorithm, that eliminates this stringent symmetry requirement. We show that device asymmetry introduces an unintentional implicit cost term into the SGD algorithm, whereas in the “Tiki-Taka” algorithm a coupled dynamical system simultaneously minimizes the original objective function of the neural network and the unintentional cost term due to device asymmetry in a self-consistent fashion. We tested the validity of this new algorithm on a range of network architectures such as fully connected, convolutional and LSTM networks. Simulation results on these various networks show that the accuracy achieved using the conventional SGD algorithm with symmetric (ideal) device switching characteristics is matched in accuracy achieved using the “Tiki-Taka” algorithm with non-symmetric (non-ideal) device switching characteristics. Moreover, all the operations performed on the arrays are still parallel and therefore the implementation cost of this new algorithm on array architectures is minimal; and it maintains the aforementioned power and speed benefits. These algorithmic improvements are crucial to relax the material specification and to realize technologically viable resistive crossbar arrays that outperform digital accelerators for similar training tasks.
Highlights
The presence of this additional term entails poor training results for non-symmetric devices as the system is in competition with minimizing the objection function of the neural network against this unintentional cost term. In this new technique we introduce a coupled dynamical system that simultaneously minimizes the objective function of the original stochastic gradient descent (SGD) algorithm as well as the unintentional cost term due to device asymmetry in a self-consistent fashion
To test the validity of the proposed “Tiki-Taka” algorithm we performed deep neural networks (DNN) training simulations on three different network architectures: (1) FCN-MNIST – a fully connected network trained on MNIST dataset, (2) CNN-MNIST – LeNet5 like convolutional neural network trained on MNIST dataset, and (3) LSTM Network on War and Peace Dataset (LSTM-WP) – a doubly stacked LSTM network trained on Leo Tolstoy’s War and Peace (WP) novel
It was shown that a very tight symmetry requirement is needed to achieve training accuracies comparable to the ones achieved with high precision floating point numbers
Summary
In the past few years, deep neural networks (DNN) (LeCun et al, 2015) have made tremendous advances, in some cases surpassing human level performance, tackling challenging problems such as speech recognition (Hinton et al, 2012; Ravanelli et al, 2017), natural language processing (Collobert et al, 2012; Jozefowicz et al, 2016), image classification (Krizhevsky et al, 2012; He et al, 2015a,b; Chen et al, 2017), and machine translation (Wu, 2016). Alternative to digital approaches, resistive cross-point device arrays are proposed to further increase the throughput and energy efficiency of the overall system by performing the vectormatrix multiplications in the analog domain. As shown empirically (Agarwal et al, 2016a; Gokmen and Vlasov, 2016; Gokmen et al, 2017), a key requirement is that these analog resistive devices must change conductance symmetrically when subjected to positive or negative voltage pulse stimuli This requirement differs significantly from those needed for memory elements and accomplishing such symmetrically switching analog devices is a difficult task. CMOS only (Li et al, 2018) and CMOS assisted solutions in tandem with existing memory device technologies (Ambrogio et al, 2018) are considered but introduce an overhead of making the cross-point element increasingly more complex
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
