Cost effective Tanh activation function circuits based on fast piecewise linear logic

Abstract

In this paper, an approximate 16-bit non-linear Tanh function circuit design is proposed for activation function in neural networks and digital signal processing, which is based on piecewise linear calculation. By sacrificing acceptable accuracy in computation, the proposed logic circuit structure can furtherly improve the overall performance of the design, with marked reduction in delay, area and power consumption. Linear calculation segments with simple shift-and-add logic architectures are adopted for easy hardware implementation. Additionally, tailored bias tuning and approximate accumulator design make a fine balance between complexity and error distribution. The mean absolute error (MAE) of the entire design is 0.0049. Compared with other state-of-art works in 90 nm CMOS process, the area of the circuit is 582μm2, and the energy is 0.0996 mW/GHz at 0.71 GHz. Moreover, the area-delay product is reduced by over 79.7% and the energy-delay product is reduced by over 42.4%. In the evaluation of its suitability in large applications, the accurate Tanh circuit and the proposed Tanh circuit are adopted in two neural networks to classify the Fashion-MNIST data and emotional EEG data. When the model converges, the accuracies of the test set only differs with a maximum of 0.2% and 0.3%.