Accurate deep neural network inference using computational phase-change memory

Vinay Joshi,Bipin Rajendran,Evangelos Eleftheriou,Manuel Le Gallo,Abu Sebastian,S R Nandakumar,Irem Boybat,Christophe Piveteau,Simon Haefeli,Martino Dazzi

doi:10.1038/s41467-020-16108-9

Abstract

In-memory computing using resistive memory devices is a promising non-von Neumann approach for making energy-efficient deep learning inference hardware. However, due to device variability and noise, the network needs to be trained in a specific way so that transferring the digitally trained weights to the analog resistive memory devices will not result in significant loss of accuracy. Here, we introduce a methodology to train ResNet-type convolutional neural networks that results in no appreciable accuracy loss when transferring weights to phase-change memory (PCM) devices. We also propose a compensation technique that exploits the batch normalization parameters to improve the accuracy retention over time. We achieve a classification accuracy of 93.7% on CIFAR-10 and a top-1 accuracy of 71.6% on ImageNet benchmarks after mapping the trained weights to PCM. Our hardware results on CIFAR-10 with ResNet-32 demonstrate an accuracy above 93.5% retained over a one-day period, where each of the 361,722 synaptic weights is programmed on just two PCM devices organized in a differential configuration.

Highlights

In-memory computing using resistive memory devices is a promising non-von Neumann approach for making energy-efficient deep learning inference hardware
We focus on the ResNet convolutional neural network (CNN) architecture, and introduce a number of techniques that allow us to achieve a classification accuracy of 93.7% on the CIFAR-10 dataset and a top-1 accuracy of 71.6% on the ImageNet benchmark after mapping the trained weights to phase-change memory (PCM) synapses
The strategies developed in this study allow us to achieve the highest accuracies reported so far with analog resistive memory on the CIFAR-10 and ImageNet benchmarks with residual networks close to their original implementation[24]

Summary

Introduction

In-memory computing using resistive memory devices is a promising non-von Neumann approach for making energy-efficient deep learning inference hardware. Both charge-based storage devices, such as Flash memory[6], and resistance-based (memristive) storage devices, such as metal-oxide resistive randomaccess memory[7,8,9,10] and phase-change memory (PCM)[11,12,13,14] are being investigated for this In this approach, the network weights are encoded as the analog charge state or conductance state of these devices organized in crossbar arrays, and the matrix-vector multiplications during inference can be performed in-situ in a single time step by exploiting Kirchhoff’s circuit laws. A more practical approach would be to have a single custom generic training algorithm run entirely in software, which would make the network immune to most of the hardware non-idealities, but at the same time would require only very little knowledge about the specific hardware it will be deployed on In this way, the model would have to be trained only once and could be deployed on a multitude of different chips. We discuss our training approach with respect to other methods and quantify the tradeoffs in terms of accuracy and ease of training

Methods

Results

Conclusion