Parametric rectified nonlinear unit (PRenu) for convolution neural networks

Abstract

Activation function unit is an extremely important part of convolution neural networks; it is the nonlinear transformation that we do over the input data. Using hidden layer incorporating with a well-chosen activation function improves both the accuracy and the CNN convergence speed. This paper proposes a parametric rectified nonlinear function unit (PRenu). The proposed activation function is nearly similar to Relu. It returns $$x-\alpha \log (x+1)$$ for positive values ( $$\alpha $$ is between 0 and 1) and zero for negative parts. In contrast to Relu that returns the same received gradient for all positive values in its back-propagation, the PRenu multiplies it by values between $$1-\alpha $$ and 1 depending on the value with which each neuron was involved. The PRenu has been tested on three datasets: CIFAR-10, CIFAR-100 and Oxflower17, and compared to the activation function Relu. The experimental results show that using the proposed activation function PRenu, the CNN convergence is faster and the accuracy is also improved.