A Note on the Exact Relation Between Mixture Likelihood and Entropy

Abstract

It is interesting to note that the expected value of the log likelihood function is entropy. This note shows that there is an exact relationship between the mixture log likelihood function (ln LM) and the sum of the mixing distribution entropy (HM) and the mixture density entropy (HD). Ln LM is seen as a function exactly of four Shannon entropies, each a unique measure of uncertainty. This method, known as mixtures of linear models (MLM), is a form of empirical Bayes which uses a non-informative uniform prior and generates both confidence intervals and p-values which clinicians and regulatory agencies can use to evaluate scientific evidence. An example based on allergic rhinitis symptoms scores are given and show how easy it is to assess the fit of the model and evaluate the results of the trial.