Behavior Of Neural Networks Research Articles

Normalization effects on deep neural networks

We study the effect of normalization on the layers of deep neural networks of feed-forward type. A given layer $ i $ with $ N_{i} $ hidden units is allowed to be normalized by $ 1/N_{i}^{\gamma_{i}} $ with $ \gamma_{i}\in[1/2,1] $ and we study the effect of the choice of the $ \gamma_{i} $ on the statistical behavior of the neural network's output (such as variance) as well as on the test accuracy on the MNIST data set. We find that in terms of variance of the neural network's output and test accuracy the best choice is to choose the $ \gamma_{i} $'s to be equal to one, which is the mean-field scaling. We also find that this is particularly true for the outer layer, in that the neural network's behavior is more sensitive in the scaling of the outer layer as opposed to the scaling of the inner layers. The mechanism for the mathematical analysis is an asymptotic expansion for the neural network's output. An important practical consequence of the analysis is that it provides a systematic and mathematically informed way to choose the learning rate hyperparameters. Such a choice guarantees that the neural network behaves in a statistically robust way as the $ N_i $ grow to infinity.

Abstracting Uncertain Knowledge: Case for Neural Nets Application

ABSTRACTNeural networks have been used successfully in practical applications where human expertise exists but no clear rules are known. There is a more difficult case when the acquisition of labelled data points is very expensive, such as in labelling of ground data to match satellite images in geographic information systems.In fact, the dependence of neural networks on large volumes of training data result in the neural solution, producing more inconsistent results over a number of trials using the same data, but different initialisations of the weights.We present our method of generating IF-THEN rules expressing the trained neural network's behaviour. By using the causal index on characteristic input patterns, we produce a list of inputs which were significant in reaching the decision made and a well-ordered sequence of rules governing this decision. This method correctly produced rules for 94% of the decisions made by a sample network.The principle of selecting the next most likely decision (that the ...