Maximum entropy inference for mixed continuous-discrete variables

Abstract

We represent knowledge by probability distributions of mixed continuous and discrete variables. From the joint distribution of all items, one can compute arbitrary conditional distributions, which may be used for prediction. However, in many cases only some marginal distributions, inverse probabilities, or moments are known. Under these conditions, a principle is needed to determine the full joint distribution of all variables. The principle of maximum entropy (Jaynes, Phys Rev 1957;106:620–630 and 1957;108:171–190; Jaynes, Probability Theory—The Logic of Science, Cambridge, UK: Cambridge University Press, 2003; Haken, Synergetics, Berlin: Springer-Verlag, 1977; Guiasu and Shenitzer, Math Intell 1985;117:83–106) ensures an unbiased estimation of the full multivariate relationships by using only known facts. For the case of discrete variables, the expert shell SPIRIT implements this approach (cf. Rödder, Artif Intell 2000;117:83–106; Rödder and Meyer, in Proceedings of the 12th Conference on Uncertainty in Artificial Intelligence, San Francisco, CA, 2006; Rödder et al., Logical J IGPL 2006;14(3):483–500). In this paper, the approach is generalized to continuous and mixed continuous-discrete distributions and applied to the problem of credit scoring. © 2010 Wiley Periodicals, Inc.