Error models of finite word length arithmetic in CNN accelerator design

Abstract

Convolution Neural Network (CNN) is a state of the art machine learning algorithm. For CNN accelerator implementations, fixed-point and floating-point are two typical numeric representations. Because of the effects of rounding, reducing the word length would save the hardware and the power overheads while sacrificing the computation accuracy. The inherent robustness of neural network makes it possible to maintain the classification accuracy with very limited word length. Therefore, for the CNN accelerator designs, the primary issue is to determine the optimal arithmetic and the associated word length. In this paper, we developed the analytical error models to investigate the finite word length impacts of fixed-point and floating-point arithmetic in the test phase of CNN, respectively. It is revealed that the rounding errors are accumulated during the layer-wise convolution. Therefore, with the augment of network scale, the required word lengths for fixed-point and floating-point are both increased.