Occam’s Razor for Parametric Families and Priors on the Space of Distributions

Abstract

I define the razor, a natural measure of the complexity of a parametric family of distributions relative to a given true distribution. I show that empirical approximations of this quantity may be used to implement parsimonious inference schemes that favour simple models. In particular, the razor is seen to give finer classifications of model families than the Minimum Description Length principle as advocated by Rissanen. In a certain strong sense it is shown that the logarithm of the Bayesian posterior probability of a model family given a collection of data converges in the large sample limit to the logarithm of the razor of the family. This provides the most accurate asymptotics to date for Bayesian parametric inference. These results are derived by treating parametric families as manifolds embedded in the space of probability distributions. In the course of deriving a suitable integration measure on such manifolds, it is shown that, in a certain sense, a uniform prior on the space of probability distributions would induce a Jeffreys’ Prior on the parameters of a parametric family of distributions.