Chernoff-Type Concentration of Empirical Probabilities in Relative Entropy

Abstract

We study the relative entropy of the empirical probability vector with respect to the true probability vector in multinomial sampling of k categories, which, when multiplied by sample size n, is also the log-likelihood ratio statistic. We generalize a recent result and show that the moment generating function of the statistic is bounded by a polynomial of degree n on the unit interval, uniformly over all true probability vectors. We characterize the family of polynomials indexed by (k,n) and obtain explicit formulae. Consequently, we develop Chernoff-type tail bounds, including a closed-form version from a large sample expansion of the bound minimizer. Our bound dominates the classic method-of-types bound and is competitive with the state of the art. We demonstrate with an application to estimating the proportion of unseen butterflies.