Abstract
Information Bottleneck-based methods use mutual information as a distortion function in order to extract relevant details about the structure of a complex system by compression. One of the approaches used to generate optimal compressed representations is by annealing a parameter. In this manuscript we present a common framework for the study of annealing in information distortion problems. We identify features that should be common to any annealing optimization problem. The main mathematical tools that we use come from the analysis of dynamical systems in the presence of symmetry (equivariant bifurcation theory). Through the compression problem, we make connections to the world of combinatorial optimization and pattern recognition. The two approaches use very different vocabularies and consider different problems to be “interesting”. We provide an initial link, through the Normalized Cut Problem, where the two disciplines can exchange tools and ideas.
Highlights
Our goal in this paper is to investigate the mathematical structure of Information Distortion methods
We will concentrate on the annealing method applied to two different functions: the Information Bottleneck cost function [1] and the Information Distortion function [2]
We show that the eigenvector corresponding to this phase transition solves the Approximate Normalized Cut problem for some graphs with vertices corresponding to elements of Y
Summary
Our goal in this paper is to investigate the mathematical structure of Information Distortion methods. We close by introducing the optimization problems we will study Both approaches attempt to characterize a system of interest (X, Y ) defined by a probability p(X, Y ) by quantizing (discretizing) one of the variables (Y here) into a reproduction variable T with few elements. We will explain all of the details in the main text, but we want to sketch the basic idea of the annealing approach here Since both functions IH(T |Y ) and −I (T ; Y ) are concave, when β = 0, both problems (1) and (2) admit a homogeneous solution q(t|y) = 1/N , where N is the number of elements in T. It is this process and its phase transitions that we consider in this contribution
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
