A Cost-Efficient Approximate Dynamic Ranged Multiplication and Approximation-Aware Training on Convolutional Neural Networks

Hyunjin Kim,Alberto A Del Barrio

doi:10.1109/access.2021.3117148

Abstract

This paper proposes a low-cost approximate dynamic ranged multiplier and describes its use during the training process on convolutional neural networks (CNNs). It has been noted that the approximate multiplier can be used in the convolution of CNN’s forward path. However, in CNN inference on a post-training quantization with a pre-trained model, erroneous convolution output from highly approximate multipliers significantly degrades performance. On the other hand, with the CNN model based on an approximate multiplier, the approximation-aware training process can optimize its learnable parameters, producing better classification results considering the approximate hardware. We analyze the error distribution of the approximate dynamic ranged multiplication and characterize it in order to find the most suitable approximate multiplier design. Considering the effects of normalizing the biased convolution outputs, a low standard deviation of relative errors with respect to the multiplication outputs leads to a negligible accuracy drop. Based on these facts, the hardware costs of the proposed multiplier can be further reduced by adopting the partial products’ inaccurate compression, truncated input fraction, and reduced-width multiplication output. When the proposed approximate multiplier is applied to the residual convolutional neural networks for the CIFAR-100 and Tiny-ImageNet datasets, the accuracy drops of the approximation-aware training results are negligible compared with those using 32-bit floating-point CNNs.

Highlights

In the implementation of machine learning, deep neural networks are used in various fields
Del Barrio: Cost-Efficient Approximate Dynamic Ranged Multiplication model is optimized for achieving better classification results, which is denoted as the approximation-aware training
We propose an approximate dynamic ranged multiplier for performing convolutions on convolutional neural networks (CNNs) and introduce an approximation-aware training method by simulating the approximation in the training stage

Summary

INTRODUCTION

In the implementation of machine learning, deep neural networks are used in various fields. A. Del Barrio: Cost-Efficient Approximate Dynamic Ranged Multiplication model is optimized for achieving better classification results, which is denoted as the approximation-aware training. CNNs using the 8-bit fixed-point format [11] binarized CNN [12], [13], and log-based CNN [14], [15] provide highly quantized models for low-cost CNN implementation. These models show significant accuracy drops in CNN inference. When training a CNN with our approximate dynamic ranged multiplier, the biased average relative error of the approximate multiplier is not critical in the performance of trained models. The experimental results in the CNN training will be shown

PRELIMINARIES

APPROXIMATION IN DYNAMIC RANGED MULTIPLIER

EXPERIMENTS WITH CNN TRAINING

TRAINING ON CIFAR DATASET

Findings

CONCLUSION