Estimating probability density functions using a combined maximum entropy moments and Bayesian method. Theory and numerical examples

Abstract

Estimating the probability density function (pdf) from a limited sample of data is a challenging data analysis problem. Furthermore, determining which pdf best describes the available data involves an extra layer of complexity to the analysis, which if ignored, can have considerable consequences.We propose a combined maximum entropy (MaxEnt) moments and Bayesian model selection method to address this problem. The MaxEnt moments component is used to formulate a set of possible pdf models, each constrained by a different set of moments and parameterised by a set of Lagrangian multipliers. The Bayesian model selection component makes an inference about the most probable model, from the set of MaxEnt moment models. The structure of the prior pdf for the Lagrangian multipliers is determined from an expansion of the free energy functional for each MaxEnt model, and corresponding hyperparameters are calculated empirically.Numerical experiments were used to test the proposed method on samples taken from Gaussian and (more complex) non-Gaussian distributions, over a range of sample sizes. The results clearly demonstrate that the method can discriminate between simple and complex MaxEnt models for sample sizes approximately greater than 60. Our results demonstrate that MaxEnt and Bayesian methods are complementary. More critically, Bayesian inference is necessary when a set of competing MaxEnt models can be derived for a single dataset from a range of assumptions.