Hessian-driven unequal protection of DNN parameters for robust inference

Abstract

Listen

Translate article

This paper presents an algorithmic approach to design reliable deep neural networks (DNN) in the presence of stochastic variations in the network parameters induced by process variations in the bit-cells in a processing-in-memory (PIM) architecture. We propose and derive a Hessian based sensitivity metric that can be computed without computing or storing the full Hessian to identify and protect the important network parameters while allowing large variations in unprotected parameters. Experiments on modern DNNs like ResNet, MobileNetv2, DenseNet on CIFAR10 demonstrates that by shielding only a small (1% -- 5%) fraction of parameters one can achieve less than 1% accuracy degradation even under large (50%) stochastic variations in other parameters.