The present article derives the minimal number N of observations needed to approximate a Bayesian posterior distribution by a Gaussian. The derivation is based on an invariance requirement for the likelihood <img src=image/13422860_01.gif>. This requirement is defined by a Lie group that leaves the <img src=image/13422860_01.gif> unchanged, when applied both to the observation(s) <img src=image/13422860_05.gif> and to the parameter <img src=image/13422860_02.gif> to be estimated. It leads, in turn, to a class of specific priors. In general, the criterion for the Gaussian approximation is found to depend on (i) the Fisher information related to the likelihood <img src=image/13422860_01.gif>, and (ii) on the lowest non-vanishing order in the Taylor expansion of the Kullback-Leibler distance between <img src=image/13422860_01.gif> and <img src=image/13422860_03.gif>, where <img src=image/13422860_04.gif> is the maximum-likelihood estimator of <img src=image/13422860_02.gif>, given by the observations <img src=image/13422860_05.gif>. Two examples are presented, widespread in various statistical analyses. In the first one, a chi-squared distribution, both the observations <img src=image/13422860_05.gif> and the parameter <img src=image/13422860_02.gif> are defined all over the real axis. In the other one, the binomial distribution, the observation is a binary number, while the parameter is defined on a finite interval of the real axis. Analytic expressions for the required minimal N are given in both cases. The necessary N is an order of magnitude larger for the chi-squared model (continuous <img src=image/13422860_05.gif>) than for the binomial model (binary <img src=image/13422860_05.gif>). The difference is traced back to symmetry properties of the likelihood function <img src=image/13422860_01.gif>. We see considerable practical interest in our results since the normal distribution is the basis of parametric methods of applied statistics widely used in diverse areas of research (education, medicine, physics, astronomy etc.). To have an analytical criterion whether the normal distribution is applicable or not, appears relevant for practitioners in these fields.
Read full abstract