A Bayesian view of model complexity

Abstract

This article addresses the problem of formally defining the ‘effective number of parameters’ in a Bayesian model which is assumed to be given by a sampling distribution and a prior distribution for the parameters. The problem occurs in the derivation of information criteria for model comparison which often trade off ‘goodness of fit’ and ‘model complexity’. It also arises in (frequentist) attempts to estimate the error variance in regression models with informative priors on the regression coefficients, for example, in smoothing. It is argued that model complexity can be conceptualized as a feature of the joint distribution of the observed variables and the random parameters and might be formally described by a measure of dependence. The universal and accurate estimation of terms of model complexity is a challenging problem in practice. Several well‐known criteria for model comparison are interpreted and discussed along these lines.