Information Bottleneck Algorithm Research Articles

Privacy-Guaranteed Two-Agent Interactions Using Information-Theoretic Mechanisms

This paper introduces a multi-round interaction problem with privacy constraints between two agents that observe correlated data. The data are assumed to have both public and private features, and the goal of the interaction is to share the public data subject to utility constraints (bounds on the distortion of public feature) while ensuring bounds on the information leakage of the private data at the other agent. The agents alternately share data with one another for a total of $K$ rounds such that each agent initiates sharing over $K/2$ rounds. The interactions are modeled as a collection of $K$ random mechanisms (mappings), one for each round. The goal is to jointly design the $K$ private mechanisms to determine the set of all achievable distortion-leakage pairs at each agent. Arguing that a mutual information-based leakage metric can be appropriate for streaming data settings, this paper: 1) determines the set of all achievable distortion-leakage tuples; 2) shows that the $K$ mechanisms allow for precisely composing the total privacy budget over $K$ rounds without loss; and 3) develops conditions under which interaction reduces the net leakage at both agents and illustrates it for a specific class of sources. The paper then focuses on log-loss distortion to better understand the effect on leakage of using a commonly used utility metric in learning theory. The resulting interaction problem leads to a non-convex sum-leakage-distortion optimization problem that can be viewed as an interactive version of the information bottleneck problem. A new merge-and-search algorithm that extends the classical agglomerative information bottleneck algorithm to the interactive setting is introduced to determine a provable locally optimal solution. Finally, the benefit of interaction under log-loss is illustrated for specific source classes, and the optimality of one-shot is proved for Gaussian sources under both mean-square and log-loss distortions constraints.

Hierarchical clustering based on the information bottleneck method using a control process

Clustering techniques aim organizing data into groups whose members are similar. A key element of these techniques is the definition of a similarity measure. The information bottleneck method provides us a full solution of the clustering problem with no need to define a similarity measure, since a variable $$X$$X is clustered depending on a control variable $$Y$$Y by maximizing the mutual information between them. In this paper, we propose a hierarchical clustering algorithm based on the information bottleneck method such that, instead of using a control variable, the different possible values of a Markov process are clustered by maximally preserving the mutual information between two consecutive states of the Markov process. These two states can be seen as the input and the output of an information channel that is used as a control process, similarly to how the variable $$Y$$Y is used as a control variable in the original information bottleneck algorithm. We present both agglomerative and divisive versions of our hierarchical clustering approach and two different applications. The first one, to quantize an image by grouping intensity bins of the image histograms, is tested on synthetic, photographic and medical images and compared with hand-labelled images, hierarchical clustering using Euclidean distance and non-negative matrix factorization methods. The second one, to cluster brain regions by grouping them depending on their connectivity, is tested on medical data. In all the applications, the obtained results demonstrate the efficacy of the method in getting clusters with high mutual information.

Optimal Kullback–Leibler Aggregation via Information Bottleneck

In this paper, we present a method for reducing a regular, discrete-time Markov chain (DTMC) to another DTMC with a given, typically much smaller number of states. The cost of reduction is defined as the Kullback-Leibler divergence rate between a projection of the original process through a partition function and a DTMC on the correspondingly partitioned state space. Finding the reduced model with minimal cost is computationally expensive, as it requires an exhaustive search among all state space partitions, and an exact evaluation of the reduction cost for each candidate partition. Our approach deals with the latter problem by minimizing an upper bound on the reduction cost instead of minimizing the exact cost. The proposed upper bound is easy to compute and it is tight if the original chain is lumpable with respect to the partition. Then, we express the problem in the form of information bottleneck optimization, and propose using the agglomerative information bottleneck algorithm for searching a suboptimal partition greedily, rather than exhaustively. The theory is illustrated with examples and one application scenario in the context of modeling bio-molecular interactions.

Using local density information to improve IB algorithms

The Information Bottleneck principle provides a systematic method to extract relevant features from complex data sets, and it models features extraction as data compression and quantifies the relevance of extracted feature by how much information it preserved about a specified feature. How to construct an optimal solution to IB remains a problem. The current Information Bottleneck (IB) algorithms only utilize the information between element pairs, and ignore the information among the neighborhood of elements. This is one of the major reasons for most IB algorithms’ failure to preserve as much relative information as possible, which further limits IB applicability in many areas. In this paper, we present the concept of density connectivity component, by which the information loss among the neighbors of an element, rather than the information loss between paired elements, can be considered. Then, we introduce this concept into the current agglomerative IB algorithm (aIB) and sequential IB algorithm (sIB), and propose two density-based IB algorithms, DaIB and DsIB. The experiment results on the benchmark data sets indicate that the DaIB and DsIB algorithm can preserve more relevant information and achieve higher precision than the aIB and sIB algorithm, respectively.