Abstract
When comparing discrete probability distributions, natural measures of similarity are not ? p distances but rather are information divergences such as Kullback-Leibler and Hellinger. This paper considers some of the issues related to constructing small-space sketches of distributions in the data-stream model, a concept related to dimensionality reduction, such that these measures can be approximated from the sketches. Related problems for ? p distances are reasonably well understood via a series of results by Johnson and Lindenstrauss (Contemp. Math. 26:189---206, 1984), Alon et al. (J. Comput. Syst. Sci. 58(1):137---147, 1999), Indyk (IEEE Symposium on Foundations of Computer Science, pp. 202---208, 2000), and Brinkman and Charikar (IEEE Symposium on Foundations of Computer Science, pp. 514---523, 2003). In contrast, almost no analogous results are known to date about constructing sketches for the information divergences used in statistics and learning theory. Our main result is an impossibility result that shows that no small-space sketches exist for the multiplicative approximation of any commonly used f-divergences and Bregman divergences with the notable exceptions of ? 1 and ? 2 where small-space sketches exist. We then present data-stream algorithms for the additive approximation of a wide range of information divergences. Throughout, our emphasis is on providing general characterizations.
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.