On the Probabilistic Interpretation of Neural Network Behavior

Abstract

Recent probabilistic interpretations of neural network models have suggested the formulaton of network operations in information-theoretic terms. In these interpretations, the neural network developes an assumed probability density function which represents its assumptions on the environment. Using a set of hypotheses, this probability density functon is shown to maintain an exponential relationship wth an energy-like functon that the network tends to minimize. The purpose of this note is to obtain this probability density function through Shannon's dervation of the entropy measure and Jaynes' maximum entropy principle. The main conclusion is that the neural network assumes the worst case (i.e. most uncertain or maximum-entropy) probability density function for the unknown environment.