Fully probabilistic design of hierarchical Bayesian models

Abstract

The minimum cross-entropy principle is an established technique for design of an unknown distribution, processing linear functional constraints on the distribution. More generally, fully probabilistic design (FPD) chooses the distribution—within the knowledge-constrained set of possible distributions—for which the Kullback-Leibler divergence to the designer’s ideal distribution is minimized. These principles treat the unknown distribution deterministically. In this paper, fully probabilistic design is applied to hierarchical Bayesian models for the first time, yielding optimal design of a (possibly nonparametric) stochastic model for the unknown distribution. This equips minimum cross-entropy and FPD distributional estimates with measures of uncertainty. It enables robust choice of the optimal model, as well as randomization of this choice. The ability to process non-linear functional constraints in the constructed distribution significantly extends the applicability of these principles. Currently available FPD procedures for (a) merging of external knowledge, (b) approximate learning and stabilized forgetting, (c) decision strategy design, and (d) local adaptive control design, are unified for the first time via the hierarchical FPD framework of this paper.