On convergence of information contained in quantized observations

Abstract

By the well-known information processing theorem, quantizations of observations reduce values of convex information functionals such as the information divergence, Fisher information, or Shannon information. This paper deals with the convergence of the reduced values of these functionals to their original unreduced values for various sequences P/sub n/ of partitions of the observation space. There is extensive literature dealing with this convergence when the partitions are nested in the sense P/sub n/ /spl sub/ P/sub n+1/ for all n. A systematic study for nonnested partitions P/sub n/, often considered in the literature, seems to be missing. This paper tries to partially fill this gap. It proves the convergence for the most common types of partitions. The results are formulated for generalized information divergences (Csiszar (1963, 1967, 1973) divergences), generalized Fisher information, and generalized Shannon information. Their applicability is illustrated on the Barron (1992) estimators of probability distributions.