Biased ReLU neural networks

Abstract

Neural networks (NN) with rectified linear units (ReLU) have been widely implemented since 2012. In this paper, we describe an activation function called the biased ReLU neuron (BReLU), which is similar to the ReLU. Based on this activation function, we propose the BReLU NN (BRNN). The structure of the BRNN is similar to that of the ReLU network. However, the difference between the two is that the BReLU introduces several biases for each input variable. This allows the BRNN to divide the input space into a greater number of linear regions and improve network flexibility. The BRNN parameters to be estimated are the weight matrices and the bias parameters of the BReLU neurons. The weights are obtained using the backpropagation method. Moreover, we propose a method to compute the bias parameters of the BReLU neurons. In this method, batch normalization is applied to the BRNN, and the variance and mean of the input variables are obtained. Based on these two parameters, the bias parameters are estimated. In addition, we investigate the flexibility of the BRNN. Specifically, we study the number of linear regions and provide the upper bound for the maximum number of linear regions. The results indicate that for the same input dimension, the BRNN divides the input space into a greater number of linear regions than the ReLU network. This explains to a certain extent why the BRNN has the superior approximation ability. Experiments are carried out using five datasets, and the results verify the effectiveness of the proposed method.