Are higher order statistics better for maximum entropy prediction in neural networks?

Abstract

Summary form only given, as follows. A neural network that generates probabilities of future events from its current neuronal states is investigated. It was hypothesized that neurons can compute a probability based on statistics stored at synapses using the maximum entropy inference method. In particular, the authors investigated the question of what kinds of statistics should be stored at synapses in order to make better predictions; lower or higher order statistics. To evaluate predictions based on two different sets of statistics, low-order statistics versus high-order statistics. Monte-Carlo simulations were run for a simple model network, and a performance measure, i.e., relative entropy, was calculated as an average across a large number of different neuronal environments. The simulations indicate that the lower-order statistics seem to make better predictions than the higher-order statistics for most of the environments in which neurons compute predictions. Implications of the results are considered in relation to the information processing in the brain. >