Monotonic Convergence in an Information-Theoretic Law of Small Numbers

Abstract

An increasing to the maximum result analogous to the entropic central limit theorem (Barron 1986; Artstein et 2004) is obtained in the discrete setting. This involves the thinning operation and a Poisson limit. Monotonic convergence in relative entropy is established for general discrete distributions, while monotonic increase of Shannon entropy is proved for the special class of ultra-log-concave distributions. Overall we extend the parallel between the information-theoretic central limit theorem and law of small numbers explored by Kontoyiannis et (2005) and Harremoes et al. (2007, 2008). Ingredients in the proofs include convexity, majorization, and stochastic orders.