Point estimation using the Kullback-Leibler loss function and MML

Abstract

We consider the problem of point estimation of an unknown continuous parameter within a Bayesian setting. Given data-space X, data α ∈ X, likelihood function p(α|θ) and prior h(θ), we estimate θ using a loss function, C(θ,?). We are interested in the case where the cost function is unknown at the time of point estimation. Invariance and other properties of the Kullback-Leibler distance lead us to consider it as a 'reference' loss function. This involves finding the? which minimises the expected Kullback-Leibler distance: Formula math. where r(α) = √ θ h(θ)p(α|θ)dθ is the marginal probability of the data. Rewriting: Formula math. The second term is not dependent on? and can be omitted from the expression to be minimized. Swapping the order of integration, Formula math. The term π(θ|α) is the posterior probability of the parameter θ given the data.r: and the term in square brackets is prob(y|x), the probability of future data y given our current data. x. Given our current data, x, and parameter estimate,?, equation (1) gives the expected log-likelihood of future data, y. We consider a minimum Expected Kullback-Leibler (EKL) estimate, the value of? which minimises the expression in Equation (1). Analytical minimisation of this expression (1) is relatively difficult for all but the simplest of cases. However, we can numerically generate simulated future data, y, by generating θ from the posterior (in θ given x) and then generating y from p(y|θ). The maximum likelihood estimate of this would-be future sample will converge in the limit to our desired minimum EKL estimate. A related invariant technique is the information-theoretic Minimum Message Length (MML) method [WF87], which entails the minimisation of a two-part message transmitting a hypothesis, θ, and the data, x, in light of θ. The MML estimator is a quadratic approximation to the Strict MML (SMML) estimator [WB75, WF87], which maps partitions of the discrete data-space to estimator values. Like the Kullback-Leibler distance. the SMML and MML estimators are invariant under parameter transformations. The MML estimator has been shown to return a relatively small Kullback-Leibler distance from the true distribution for various models, including circular distributions[WD3] and factor analysis. The MML estimator minimises a two-part message length of transmitting θ and the current data, x, in light of θ. The minimum EKL estimator minimises the expected length of the transmission of future data. We now present and argument as to why these two methods are similar. First, an approximation to SMML called Fairly Strict MML (FSMML) maps regions from the parameter space to point estimates. These regions, or coding blocks, can then be used to generate synthetic data by convolving h(θ) with p(x|θ) over the coding block. Taking the Maximum Likelihood estimate for this synthetic data gives the FSMML estimate.?. for the coding block. Minimum EKL takes a prior, h(θ), and data, x, to produce a posterior, π(θ|x). Whether we analytically convolve the posterior with p(x|θ) or whether we numerically sample a θ from the posterior and then a datum, y, from p(x|θ), we obtain a population of expected future data. The minimum EKL estimator is the maximum likelihood estimator on the expected future data. These two methods are very similar, and only differ insofar as the minimum EKL uses the posterior whereas FSMML uses the prior over the coding block. Given that we do not know exactly where our coding block regions might be in FSMML, we can average the location of the coding. This leads to something very akin to the posterior.