Information Bottleneck Approach Research Articles

Data-Efficient Dimensionality Reduction and Surrogate Modeling of High-Dimensional Stress Fields

Abstract Tensor datatypes representing field variables like stress, displacement, velocity, etc., have increasingly become a common occurrence in data-driven modeling and analysis of simulations. Numerous methods [such as convolutional neural networks (CNNs)] exist to address the meta-modeling of field data from simulations. As the complexity of the simulation increases, so does the cost of acquisition, leading to limited data scenarios. Modeling of tensor datatypes under limited data scenarios remains a hindrance for engineering applications. In this article, we introduce a direct image-to-image modeling framework of convolutional autoencoders enhanced by information bottleneck loss function to tackle the tensor data types with limited data. The information bottleneck method penalizes the nuisance information in the latent space while maximizing relevant information making it robust for limited data scenarios. The entire neural network framework is further combined with robust hyperparameter optimization. We perform numerical studies to compare the predictive performance of the proposed method with a dimensionality reduction-based surrogate modeling framework on a representative linear elastic ellipsoidal void problem with uniaxial loading. The data structure focuses on the low-data regime (fewer than 100 data points) and includes the parameterized geometry of the ellipsoidal void as the input and the predicted stress field as the output. The results of the numerical studies show that the information bottleneck approach yields improved overall accuracy and more precise prediction of the extremes of the stress field. Additionally, an in-depth analysis is carried out to elucidate the information compression behavior of the proposed framework.

Task-Oriented Communication for Graph Data: A Graph Information Bottleneck Approach

Task-Oriented Communication for Graph Data: A Graph Information Bottleneck Approach

Quantifying the Relevance of Long-Range Forces for Crystal Nucleation in Water.

Understanding nucleation from aqueous solutions is of fundamental importance in a multitude of fields, ranging from materials science to biophysics. The complex solvent-mediated interactions in aqueous solutions hamper the development of a simple physical picture, elucidating the roles of different interactions in nucleation processes. In this work, we make use of three complementary techniques to disentangle the role played by short- and long-range interactions in solvent-mediated nucleation. Specifically, the first approach we utilize is the local molecular field (LMF) theory to renormalize long-range Coulomb electrostatics. Second, we use well-tempered metadynamics to speed up rare events governed by short-range interactions. Third, the deep learning-based State Predictive Information Bottleneck approach is employed in analyzing the reaction coordinate of the nucleation processes obtained from the LMF treatment coupled with well-tempered metadynamics. We find that the two-step nucleation mechanism can largely be captured by the short-range interactions, while the long-range interactions further contribute to the stability of the primary crystal state under ambient conditions. Furthermore, by analyzing the reaction coordinate obtained from the combined LMF-metadynamics treatment, we discern the fluctuations on different time scales, highlighting the need for long-range interactions when accounting for metastability.

Gated information bottleneck for generalization in sequential environments

Deep neural networks suffer from poor generalization to unseen environments when the underlying data distribution is different from that in the training set. By learning minimum sufficient representations from training data, the information bottleneck (IB) approach has demonstrated its effectiveness to improve generalization in different AI applications. In this work, we propose a new neural network-based IB approach, termed gated information bottleneck (GIB), that dynamically drops spurious correlations and progressively selects the most task-relevant features across different environments by a trainable soft mask (on raw features). Using the recently proposed matrix-based Rényi’s \(\alpha \)-order mutual information estimator, GIB enjoys a simple and tractable objective, without any variational approximation or distributional assumption. We empirically demonstrate the superiority of GIB over other popular neural network-based IB approaches in adversarial robustness and out-of-distribution detection. Meanwhile, we also establish the connection between IB theory and invariant causal representation learning and observed that GIB demonstrates appealing performance when different environments arrive sequentially, a more practical scenario where invariant risk minimization fails.

Multi-rater label fusion based on an information bottleneck for fundus image segmentation

In the fundus image segmentation process, more than one professional rater is usually required to mark the organ in the image to reduce the error rate of diagnosis. However, it is undeniable that each rater has an independent level of expertise and experience, and there are often individual differences in annotation areas, which can produce redundant or irrelevant information. In computer vision, for feature selection of multiple annotations, the majority voting principle or the preferred annotator principle is usually used, which cannot effectively remove the noise inside the annotator and the redundant information between annotators. To solve the above problems, we studied the information bottleneck method to remove the annotation noise and extend it to multi-expert annotation tasks in the fundus image field to extract consistent information between different views. We believe that consistent information is the key information that multiple experts agree on and have directivity. To the best of our knowledge, this is the first model for extracting multi-expert consistency information via multi-view information bottlenecks. Specifically, we use a multi-view information bottleneck approach to obtain the most concise expression of each view under label supervision. In addition, we propose a novel unsupervised information bottleneck method by maximizing mutual information between multiple view representations to preserve consistent information while eliminating redundant information that is not shared between views. A large number of experiments on several public datasets prove the effectiveness of the proposed model, and its performance is superior to existing techniques.

State predictive information bottleneck.

The ability to make sense of the massive amounts of high-dimensional data generated from molecular dynamics simulations is heavily dependent on the knowledge of a low-dimensional manifold (parameterized by a reaction coordinate or RC) that typically distinguishes between relevant metastable states, and which captures the relevant slow dynamics of interest. Methods based on machine learning and artificial intelligence have been proposed over the years to deal with learning such low-dimensional manifolds, but they are often criticized for a disconnect from more traditional and physically interpretable approaches. To deal with such concerns, in this work we propose a deep learning based state predictive information bottleneck approach to learn the RC from high-dimensional molecular simulation trajectories. We demonstrate analytically and numerically how the RC learnt in this approach is connected to the committor in chemical physics and can be used to accurately identify transition states. A crucial hyperparameter in this approach is the time delay or how far into the future the algorithm should make predictions about. Through careful comparisons for benchmark systems, we demonstrate that this hyperparameter choice gives useful control over how coarse-grained we want the metastable state classification of the system to be. We thus believe that this work represents a step forward in systematic application of deep learning based ideas to molecular simulations.

Optimal prediction with resource constraints using the information bottleneck.

Responding to stimuli requires that organisms encode information about the external world. Not all parts of the input are important for behavior, and resource limitations demand that signals be compressed. Prediction of the future input is widely beneficial in many biological systems. We compute the trade-offs between representing the past faithfully and predicting the future using the information bottleneck approach, for input dynamics with different levels of complexity. For motion prediction, we show that, depending on the parameters in the input dynamics, velocity or position information is more useful for accurate prediction. We show which motion representations are easiest to re-use for accurate prediction in other motion contexts, and identify and quantify those with the highest transferability. For non-Markovian dynamics, we explore the role of long-term memory in shaping the internal representation. Lastly, we show that prediction in evolutionary population dynamics is linked to clustering allele frequencies into non-overlapping memories.

Identifying legitimate Web users and bots with different traffic profiles — an Information Bottleneck approach

Recent studies reported that about half of Web users nowadays are intelligent agents (Web bots). Many bots are impersonators operating at a very high sophistication level, trying to emulate navigational behaviors of legitimate users (humans). Moreover, bot technology continues to evolve which makes bot detection even harder. To deal with this problem, many advanced methods for differentiating bots from humans have been proposed, a large part of which relies on supervised machine learning techniques. In this paper, we propose a novel approach to identify various profiles of bots and humans which combines feature selection and unsupervised learning of HTTP-level traffic patterns to develop a user session classification model. Session clustering is performed with the agglomerative Information Bottleneck (aIB) algorithm, as well as with some other reference algorithms. The model is then used to classify new sessions to one of the profiles and to label the sessions as performed by bots or humans. An extensive experimental study, based on real server log data, demonstrates the ability of aIB clustering to distinguish user profiles and confirms high performance of the classification model in terms of accuracy, F1, recall, and precision.

Decoding Rate-Compatible 5G-LDPC Codes With Coarse Quantization Using the Information Bottleneck Method

Increased data rates and very low-latency requirements place strict constraints on the computational complexity of channel decoders in the new 5G communications standard. Practical low-density parity-check (LDPC) decoder implementations use message-passing decoding with finite precision, which becomes coarse as complexity is more severely constrained. In turn, performance degrades as the precision becomes more coarse. Recently, the information bottleneck (IB) method was used to design mutual-information-maximizing mappings that replace conventional finite-precision node computations. As a result, the exchanged messages in the IB approach can be represented with a very small number of bits. 5G LDPC codes have the so-called protograph-based raptor-like (PBRL) structure which offers inherent rate-compatibility and excellent performance. This paper extends the IB principle to the flexible class of PBRL LDPC codes as standardized in 5G. The extensions include IB decoder design for puncturing and rate-compatibility. In contrast to existing IB decoder design techniques, the proposed decoder can be used for a large range of code rates with a static set of optimized mappings. The proposed construction approach is evaluated for a typical range of code rates and bit resolutions ranging from 3 bit to 5 bit. Frame error rate simulations show that the proposed scheme always outperforms min-sum decoding algorithms and operates close to double-precision sum-product belief propagation decoding. Furthermore, alternatives to the lookup table implementations of the mutual-information-maximizing mappings are investigated.

Abnormal behavior detection in videos using deep learning

A new method for abnormal behavior detection is proposed using deep learning. Two SDAEs are utilized to automatically learn appearance feature and motion feature respectively, which are constrained in the space–time volume along dense trajectories that carry rich motion information to reduce the computational complexity. The vision words are exploited to describe behavior by the bag of words, and in order to reduce feature dimensions, the Agglomerative Information Bottleneck approach is used for vocabulary compression. An adaptive feature fusion method is adopted to enhance the discriminating power of these features. A sparse representation is exploited to abnormal behavior detection, which improve the detection accuracy. The proposed method is verified on the public dataset BEHAVE and BOSS and the results indicate the effectiveness of the proposed method.

Applying the information bottleneck to statistical relational learning

In this paper we propose to apply the Information Bottleneck (IB) approach to the sub-class of Statistical Relational Learning (SRL) languages that are reducible to Bayesian networks. When the resulting networks involve hidden variables, learning these languages requires the use of techniques for learning from incomplete data such as the Expectation Maximization (EM) algorithm. Recently, the IB approach was shown to be able to avoid some of the local maxima in which EM can get trapped when learning with hidden variables. Here we present the algorithm Relational Information Bottleneck (RIB) that learns the parameters of SRL languages reducible to Bayesian Networks. In particular, we present the specialization of RIB to a language belonging to the family of languages based on the distribution semantics, Logic Programs with Annotated Disjunction (LPADs). This language is prototypical for such a family and its equivalent Bayesian networks contain hidden variables. RIB is evaluated on the IMDB, Cora and artificial datasets and compared with LeProbLog, EM, Alchemy and PRISM. The experimental results show that RIB has good performances especially when some logical atoms are unobserved. Moreover, it is particularly suitable when learning from interpretations that share the same Herbrand base.

Discrete profile comparison using information bottleneck

Sequence homologs are an important source of information about proteins. Amino acid profiles, representing the position-specific mutation probabilities found in profiles, are a richer encoding of biological sequences than the individual sequences themselves. However, profile comparisons are an order of magnitude slower than sequence comparisons, making profiles impractical for large datasets. Also, because they are such a rich representation, profiles are difficult to visualize. To address these problems, we describe a method to map probabilistic profiles to a discrete alphabet while preserving most of the information in the profiles. We find an informationally optimal discretization using the Information Bottleneck approach (IB). We observe that an 80-character IB alphabet captures nearly 90% of the amino acid occurrence information found in profiles, compared to the consensus sequence's 78%. Distant homolog search with IB sequences is 88% as sensitive as with profiles compared to 61% with consensus sequences (AUC scores 0.73, 0.83, and 0.51, respectively), but like simple sequence comparison, is 30 times faster. Discrete IB encoding can therefore expand the range of sequence problems to which profile information can be applied to include batch queries over large databases like SwissProt, which were previously computationally infeasible.

Feature characterization in fMRI data: the Information Bottleneck approach

Clustering is a well-known technique for the analysis of Functional Magnetic Resonance Imaging (fMRI) data, whose main advantage is certainly flexibility: given a metric on the dataset, it “summarizes” the main characteristics of the data. But intrinsic to this approach are also the problem of defining correctly the quantization accuracy, and the number of clusters necessary to describe the data. The Information Bottleneck (IB) approach to vector quantization, proposed by Bialek and Tishby, addresses these difficulties: (1) it deals with an explicit trade-off between quantization and data fidelity; (2) it does so during the clustering procedure and not post hoc; (3) it takes into account the full statistical distribution of the features within the feature space and not only their most likely value; last, it is principled through an information theoretic formulation, which is relevant in many situations. In this paper, we present how to benefit from this method to analyze fMRI data. Our application is the clustering of voxels according to the magnitude of their responses to several experimental conditions. The IB quantization provides a consistent representation of the data, allowing for an easy interpretation and comparison of datasets.