Boltzmann-Shannon interaction entropy: A normalized measure for continuous variables with an application as a subsample quality metric.

Abstract

The recent introduction of geometric partition entropy offered an alternative to differential Shannon entropy for the quantification of uncertainty as estimated from a sample drawn from a one-dimensional bounded continuous probability distribution. In addition to being a fresh perspective for the basis of continuous information theory, this new approach provided several improvements over traditional entropy estimators including its effectiveness on sparse samples and a proper incorporation of the impact from extreme outliers. However, a complimentary relationship exists between the new geometric approach and the basic form of its frequency-based predecessor that is leveraged here to define an entropy measure with no bias toward the sample size. This stable normalized measure is named the Boltzmann-Shannon interaction entropy (BSIE)) as it is defined in terms of a standard divergence between the measure-based and frequency-based distributions that can be associated with the two historical figures. This parameter-free measure can be accurately estimated in a computationally efficient manner, and we illustrate its utility as a quality metric for subsampling in the context of nonlinear polynomial regression.