Abstract
AbstractRecent advances in deep neural networks have achieved higher accuracy with more complex models. Nevertheless, they require much longer training time. To reduce the training time, training methods using quantized weight, activation, and gradient have been proposed. Neural network calculation by integer format improves the energy efficiency of hardware for deep learning models. Therefore, training methods for deep neural networks with fixed point format have been proposed. However, the narrow data representation range of the fixed point format degrades neural network accuracy. In this work, we propose a new fixed point format named shifted dynamic fixed point (S-DFP) to prevent accuracy degradation in quantized neural networks training. S-DFP can change the data representation range of dynamic fixed point format by adding bias to the exponent. We evaluated the effectiveness of S-DFP for quantized neural network training on the ImageNet task using ResNet-34, ResNet-50, ResNet-101 and ResNet-152. For example, the accuracy of quantized ResNet-152 is improved from 76.6% with conventional 8-bit DFP to 77.6% with 8-bit S-DFP.
Highlights
Deep neural networks (DNNs) have shown remarkable performance on various tasks such as classification tasks [1, 2], semantic segmentation [3], and object detection [4, 5]
We propose a new fixed point format named shifted dynamic fixed point (S-DFP) to prevent accuracy degradation in quantized neural networks training
We evaluated the effectiveness of S-DFP for quantized neural network training on the ImageNet task using ResNet-34, ResNet-50, ResNet-101 and ResNet-152
Summary
Deep neural networks (DNNs) have shown remarkable performance on various tasks such as classification tasks [1, 2], semantic segmentation [3], and object detection [4, 5]. We propose a new fixed point format named shifted dynamic fixed point (S-DFP) to prevent accuracy degradation due to quantized neural networks. We evaluated the effectiveness of S-DFP for quantized neural network training on the ImageNet task using ResNet-34, ResNet-50, ResNet-101, and ResNet-152. The evaluated models can be trained using 8-bit S-DFP (S-DFP8) without any marked accuracy degradation. – We propose a training method of quantized DNN models using S-DFP8 with no marked accuracy degradation. Accuracy degradation on quantized DNN training is prevented by changing the data representation range of the weight gradient with S-DFP during weight update. The rest of this paper, the related earlier works are reviewed, and Section 3describes the proposed S-DFP format for quantized neural networks training.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
