On minimizing distortion and relative entropy

Abstract

A common approach for estimating a probability mass function w when given a prior q and moment constraints given by Aw/spl les/b is to minimize the relative entropy between w and q subject to the set of linear constraints. In such cases, the solution w is known to have exponential form. We consider the case in which the linear constraints are noisy, uncertain, infeasible, or otherwise soft. A solution can then be obtained by minimizing both the relative entropy and violation of the constraints Aw/spl les/b. A penalty parameter /spl sigma/ weights the relative importance of these two objectives. We show that this penalty formulation also yields a solution w with exponential form. If the distortion is based on an /spl lscr//sub p/ norm, then the exponential form of w is shown to have exponential decay parameters that are bounded as a function of /spl sigma/. We also state conditions under which the solution w to the penalty formulation will result in zero distortion, so that the moment constraints hold exactly. These properties are useful in choosing penalty parameters, evaluating the impact of chosen penalty parameters, and proving properties about methods that use such penalty formulations.