Class Difficulty based Mixed Precision Quantization for Low Complexity CNN Training

Abstract

Low-bit quantization of CNN training is highly needed for reducing the computational complexity of convolutional neural network (CNN) training. In CNN training, some of the classes can finish training early (reaches high accuracy in early training epochs) while other classes need more time (epochs) to finish training. This measure of training difficulty can be efficiently exploited for the mixed precision quantization to reduce the computational complexity of CNN training. In this paper, we present a training difficulty based mixed precision training approach, where easy-to-train classes are trained using low-bit quantization and the hard-to-train classes are trained using high bit quantization. The simulation results show that the proposed mixed precision training can achieve 1.33X improved compression ratio with the same accuracy compared to 8-bit (activations and weights) and 16-bit (gradients of activation and weight) uniform quantization training for ResNet-20 using the CIFAR-10 dataset.