General statistical inference by an approximate application of the maximum entropy principle

Abstract

We propose a learning method for building a general statistical inference engine, operating on discrete feature spaces. Such a model allows inference on any feature given values for the other features (or for a feature subset). Bayesian networks (BNs) are versatile tools that possess this inference capability. However, while the BN's explicit representation of conditional independencies is informative, this structure is not so easily learned. Typically, learning methods for BNs use (suboptimal) greedy search techniques. There is also a difficult issue of overfitting in these models. Alternatively, in Cheeseman (1983) proposed finding the maximum entropy (ME) joint probability mass function (pmf) consistent with arbitrary lower order probability constraints. This approach has some potential advantages over BNs. However, the huge complexity required for learning the joint pmf has severely limited the use of this approach until now. Here we propose an approximate ME method which also allows incorporation of arbitrary lower order constraints, but while retaining quite tractable learning complexity. The new method approximates the joint feature pmf (during learning) on a subgrid of the full feature space grid. Experimental results on the UC-Irvine repository reveal significant performance gains over two BN approaches: Chow and Liu's (1968) dependence trees and Herskovits and Cooper's (1991) Kutato. Several extensions of our approach are indicated.