Alternative Dirichlet Priors for Estimating Entropy via a Power Sum Functional

Tanita Botha,Johannes Ferreira,Andriette Bekker

doi:10.3390/math9131493

Tanita Botha, Johannes Ferreira + Show 1 more

Open Access

https://doi.org/10.3390/math9131493

Copy DOI

Abstract

Entropy is a functional of probability and is a measurement of information contained in a system; however, the practical problem of estimating entropy in applied settings remains a challenging and relevant problem. The Dirichlet prior is a popular choice in the Bayesian framework for estimation of entropy when considering a multinomial likelihood. In this work, previously unconsidered Dirichlet type priors are introduced and studied. These priors include a class of Dirichlet generators as well as a noncentral Dirichlet construction, and in both cases includes the usual Dirichlet as a special case. These considerations allow for flexible behaviour and can account for negative and positive correlation. Resultant estimators for a particular functional, the power sum, under these priors and assuming squared error loss, are derived and represented in terms of the product moments of the posterior. This representation facilitates closed-form estimators for the Tsallis entropy, and thus expedite computations of this generalised Shannon form. Select cases of these proposed priors are considered to investigate the impact and effect on the estimation of Tsallis entropy subject to different parameter scenarios.

Highlights

Shannon entropy and related information measures are functionals of probability and a measurement of information contained in a system that arise in information theory, machine learning and text modelling, amongst others
The paper illustrates how a Bayesian approach is applied in a multinomial-Dirichlet family setup, which allows us to obtain a posterior distribution from where explicit expressions for the Tsallis entropy can be derived, by focussing on the Product moment for the power sum functional, and assuming squared error loss
The Tsallis entropy considered in this paper, which is a popular generalised entropy, tends to Shannon entropy as α tends to 1 [14] and is given by

Summary

Introduction

Shannon entropy and related information measures are functionals of probability and a measurement of information contained in a system that arise in information theory, machine learning and text modelling, amongst others. Numerous inferential tasks rely on data-driven procedures to estimate these quantities In these settings and utilising the estimated quantities, researchers are often confronted with data arising from an unknown discrete distribution, and seek to estimate its entropy. Experimentation on diverse data sets might necessitate parameter-rich priors; this study proposes these alternative Dirichlet priors to address this potential challenge. The paper illustrates how a Bayesian approach is applied in a multinomial-Dirichlet family setup, which allows us to obtain a posterior distribution from where explicit expressions for the Tsallis entropy can be derived, by focussing on the Product moment for the power sum functional, and assuming squared error loss.

Essential Components

Alternative Dirichlet Priors

Noncentral Dirichlet Prior

Dirichlet Generator Prior

Conclusions