Analog memory-based techniques for accelerating the training of fully-connected deep neural networks (Conference Presentation)

Abstract

Crossbar arrays of resistive non-volatile memories (NVMs) offer a novel and innovative solution for deep learning tasks which are typically implemented on GPUs [1]. The highly parallel structure employed in these architectures enables fast and energy-efficient multiply-accumulate computations, which is the workhorse of most deep learning algorithms. More specifically, we are developing analog hardware platforms for acceleration of large Fully Connected (FC) Deep Neural Networks (DNNs) [1,2], where training is performed using the backpropagation algorithm. This algorithm is a supervised form of learning based on three steps: forward propagation of input data through the network (a.k.a. forward inference), comparison of the inference results with ground truth labels and backpropagation of the errors from the output to the input layer, and then in-situ weight updates. This type of supervised training has been shown to succeed even in the presence of a substantial number of faulty NVMs, relaxing yield requirements vis-a-vis conventional memory, where near 100% yield may be required [2]. We recently surveyed the use of analog memory devices for DNN hardware accelerators based on crossbar array structures and discussed design choices, device and circuit readiness, and the most promising opportunities compared digital accelerators [3]. In this presentation, we will focus on our implementation of an analog memory cell based on Phase-Change Memory (PCM) and 3-Transistor 1-Capacitor (3T1C) [4]. Software-equivalent accuracy on various datasets (MNIST, MNIST with noise, CIFAR-10, CIFAR-100) was achieved in a mixed software-hardware demonstration with DNN weights stored in real PCM device arrays as analog conductances. We will discuss how limitations from real-world non-volatile memory (NVM), such as conductance linearity and variability affects DNN training and how using two pairs of analog weights with varying significance relaxes device requirements [5, 6, 7]. Finally, we summarize all pieces needed to build an analog accelerator chip [8] and how lithography plays a role in future development of novel NVM devices.