Fast Backpropagation using Modified Sigmoidal Functions

Abstract

Backpropagation is the most commonly known algorithm for the adjustment of the weights of an Artificial Neural Network (ANN). It is well known that, in this method the partial derivatives of a criterion function with respect to the weights of a multilayer ANN are determined and the weights of the connections are adjusted pursuing a gradient descent in the weights space. At the same time, the standard function used to introduce some nonlinearity in the model is a sigmoid (or squashing function). The derivative of this function plays a relevant role in the correction process along with the actual output error and the relaxation coefficient of the procedure. Actually, the backpropagation method is a computationally efficient method to train ANNs with differentiable nonlinearities, such as sigmoidal functions. However, often the method converges very slowly, if not at all. In this paper we shall analyse a possible modification of the standard sigmoids which seem to be speeding up the training in a number of experiments we have carried out. The modification is based on analytical rather than on heuristic considerations. This correction explains why a trick used in (Fahlmann, 1988) to accelerate convergence gives good results. For the reason described we need to roughly present the backpropagation algorithm. The technique here presented allows some considerations about the choice of the learning rates as well as about the initialization of the weights in training ANNs. Although the observations made are of general interest, we will use them in problems of functions approximation: some results concerning electromagnetic identification problems are presented to substantiate the approach.