Determining probability distributions by maximum entropy and minimum cross-entropy

Abstract

The principle of maximum entropy and a generalization, the principle of minimum cross entropy, are prescriptions for solving problems of the following sort, which are encountered in a remarkable number of different fields. Namely, some system may be in any one of a given set of states; the probabilities of its being in the various states are not specified, but information about the probability distribution is available in the form of expectation values of given functions on the set of states. It is required to assign an optimal, or minimally prejudiced, set of state probabilities consistent with the given information. This paper describes the principles and presents APL functions that use the Newton-Raphson method to compute maximum-entropy and minimum-cross-entropy probability distributions for arbitrary sets of given expectation values. Example calculations are given.