Deep feedforward networks: application to pattern recognition

Abstract

Issues pertaining to learning in deep feedforward neural networks are addressed. The sensitivity of the network output is derived as a function of weight perturbation in different hidden layers. For a given hidden layer l, the output sensitivity varies inversely with current activation levels of neurons of the previous layer l-1, and the magnitude of the connection weights between layers l and l-1. Learning involves modifying weights. Relatively small connection weights (usually during initial learning phase of BP algorithm), or small neuron activation levels can increase the sensitivity of the network and make learning unstable. This problem is further aggravated as the depth of the network increases. A weight initialization strategy and a modified activation (sigmoid) function are proposed to alleviate these problems. Using this scheme, deep networks trained with the error backpropagation learning rule show substantial improvement in error curve (trajectory) control and convergence speed when applied to pattern recognition problems.