Variational Information Bottleneck Research Articles

Learning Task-Oriented Channel Allocation for Multi-Agent Communication

Benefiting from the rapid progress of wireless communication and artificial intelligence, multi-agent collaboration opens up new opportunities for various fields. To facilitate multi-agent acting as a group, effective communication plays a crucial role. Recently, many efforts based on multi-agent reinforcement learning have been made to enable effective multi-agent communication under limited bandwidth or noisy channel. However, current methods do not explore wireless resource allocation strategy explicitly. Moreover, due to ignoring task-relevant significance of information, traditional wireless resource allocation schemes may fail to guarantee the transmission efficiency and reliability for multi-agent communication. To this end, in this paper, we propose a task-oriented communication principle for multi-agent communication. We model the task-oriented channel allocation problem as a decentralized partially observable Markov decision process and propose a multi-agent reinforcement learning framework as a solution. Specifically, we design a novel variational information bottleneck to extract task-relevant information from local observation. Furthermore, a task-oriented channel allocation mechanism is developed to choose the allocation pattern with maximum expected gain. Finally, a double attention mechanism is developed to motivate the efficient utilization of task-relevant information. Experimental results show that our method can improve the effectiveness and efficiency of multi-agent communication, enhancing collaboration performance compared to baselines.

Information-Theoretic Odometry Learning

In this paper, we propose a unified information theoretic framework for learning-motivated methods aimed at odometry estimation, a crucial component of many robotics and vision tasks such as navigation and virtual reality where relative camera poses are required in real time. We formulate this problem as optimizing a variational information bottleneck objective function, which eliminates pose-irrelevant information from the latent representation. The proposed framework provides an elegant tool for performance evaluation and understanding in information-theoretic language. Specifically, we bound the generalization errors of the deep information bottleneck framework and the predictability of the latent representation. These provide not only a performance guarantee but also practical guidance for model design, sample collection, and sensor selection. Furthermore, the stochastic latent representation provides a natural uncertainty measure without the needs for extra structures or computations. Experiments on two well-known odometry datasets demonstrate the effectiveness of our method.

A novel quality‐related process monitoring method for multi‐unit industrial processes under incomplete data conditions

AbstractModern industrial processes usually consist of multiple local units. With the wide application of distributed control systems, a large amount of industrial data is collected. During data acquisition and transmission, outliers and missing data commonly exist in different local units. Under such incomplete data conditions, the detection performance of traditional quality‐related process monitoring methods may be seriously degraded. This study develops a novel quality‐related process monitoring method for multi‐unit industrial processes. Considering the existence of outliers and missing data, incomplete data preprocessing is first performed on raw process data and quality data. Then, process variables are divided into several blocks according to the process knowledge. Afterwards, the quality‐related features extracted by variational information bottleneck (VIB) are used to train deep support vector data description (DSVDD). Finally, a quality‐related process monitoring strategy is designed at both block and global levels. Experimental results on the revised Tennessee Eastman process demonstrate that the proposed method has better monitoring performance under incomplete data conditions than other state‐of‐the‐art methods.

Graph Structure Learning with Variational Information Bottleneck

Graph Neural Networks (GNNs) have shown promising results on a broad spectrum of applications. Most empirical studies of GNNs directly take the observed graph as input, assuming the observed structure perfectly depicts the accurate and complete relations between nodes. However, graphs in the real-world are inevitably noisy or incomplete, which could even exacerbate the quality of graph representations. In this work, we propose a novel Variational Information Bottleneck guided Graph Structure Learning framework, namely VIB-GSL, in the perspective of information theory. VIB-GSL is the first attempt to advance the Information Bottleneck (IB) principle for graph structure learning, providing a more elegant and universal framework for mining underlying task-relevant relations. VIB-GSL learns an informative and compressive graph structure to distill the actionable information for specific downstream tasks. VIB-GSL deduces a variational approximation for irregular graph data to form a tractable IB objective function, which facilitates training stability. Extensive experimental results demonstrate that the superior effectiveness and robustness of the proposed VIB-GSL.

Probability Distribution Representation Learning for Image-Text Cross-Modal Retrieval

In image-text cross-modal retrieval, most existing methods map the samples to point representations that relate the given sample to the specific single point and do not reflect the correlations between the given sample and all the points in semantic space, so that the semantic complexities of the samples and the local similarities between the samples are not fully exploited. To address the above issues, a probability distribution representation learning approach for image-text cross-modal retrieval is presented. Specifically, the method first incorporates lots of label information to capture the prominent features of the samples, which guides the model to construct the semantic spaces for different modalities and learn the distributions of the samples, based on variational information bottleneck. The probability densities of the learned distribution naturally reflect the correlations between the corresponding sample and all the points in semantic space. Moreover, the hinge triplet loss is introduced to align the distributions of the samples from different modalities at the semantic level, which makes the similar image-text pair have similar distributions. Finally, the learned distributions are used to represent the corresponding samples and the Bhattacharyya distance of the learned distributions to measure the similarities between the samples is leveraged, which is able to model the semantic complexities and explore the local similarities. The results of the experiments on Wikipedia and Pascal Sentence demonstrate that proposed method outperforms all of the 9 compared methods. The average mAP for the retrieval tasks of image to text and text to image evenly improves 15.0 percent points on Wikipedia and 13.6 percent points on Pascal Sentence.

MOOC Dropout Prediction Based on Multidimensional Time-Series Data

Massive open online courses have attracted millions of learners worldwide with flexible learning options. However, online learning differs from offline education in that the lack of communicative feedback is a drawback that magnifies high dropout rates. The analysis and prediction of student’s online learning process can help teachers find the students with dropout tendencies in time and provide additional help. Previous studies have shown that analyzing learning behaviors at different time scales leads to different prediction results. In addition, noise in the time-series data of student behavior can also interfere with the prediction results. To address these issues, we propose a dropout prediction model that combines a multiscale fully convolutional network and a variational information bottleneck. The model extracts multiscale features of student behavior time-series data by constructing a multiscale full convolutional network and then uses a variational information bottleneck to suppress the effect of noise on the prediction results. This study conducted multiple cross-validation experiments on KDD CUP 2015 data set. The results showed that the proposed method achieved the best performance compared to the baseline method.

Microbiome-based disease prediction with multimodal variational information bottlenecks.

Scientific research is shedding light on the interaction of the gut microbiome with the human host and on its role in human health. Existing machine learning methods have shown great potential in discriminating healthy from diseased microbiome states. Most of them leverage shotgun metagenomic sequencing to extract gut microbial species-relative abundances or strain-level markers. Each of these gut microbial profiling modalities showed diagnostic potential when tested separately; however, no existing approach combines them in a single predictive framework. Here, we propose the Multimodal Variational Information Bottleneck (MVIB), a novel deep learning model capable of learning a joint representation of multiple heterogeneous data modalities. MVIB achieves competitive classification performance while being faster than existing methods. Additionally, MVIB offers interpretable results. Our model adopts an information theoretic interpretation of deep neural networks and computes a joint stochastic encoding of different input data modalities. We use MVIB to predict whether human hosts are affected by a certain disease by jointly analysing gut microbial species-relative abundances and strain-level markers. MVIB is evaluated on human gut metagenomic samples from 11 publicly available disease cohorts covering 6 different diseases. We achieve high performance (0.80 < ROC AUC < 0.95) on 5 cohorts and at least medium performance on the remaining ones. We adopt a saliency technique to interpret the output of MVIB and identify the most relevant microbial species and strain-level markers to the model’s predictions. We also perform cross-study generalisation experiments, where we train and test MVIB on different cohorts of the same disease, and overall we achieve comparable results to the baseline approach, i.e. the Random Forest. Further, we evaluate our model by adding metabolomic data derived from mass spectrometry as a third input modality. Our method is scalable with respect to input data modalities and has an average training time of < 1.4 seconds. The source code and the datasets used in this work are publicly available.

Representation Learning for Dynamic Functional Connectivities via Variational Dynamic Graph Latent Variable Models.

Latent variable models (LVMs) for neural population spikes have revealed informative low-dimensional dynamics about the neural data and have become powerful tools for analyzing and interpreting neural activity. However, these approaches are unable to determine the neurophysiological meaning of the inferred latent dynamics. On the other hand, emerging evidence suggests that dynamic functional connectivities (DFC) may be responsible for neural activity patterns underlying cognition or behavior. We are interested in studying how DFC are associated with the low-dimensional structure of neural activities. Most existing LVMs are based on a point process and fail to model evolving relationships. In this work, we introduce a dynamic graph as the latent variable and develop a Variational Dynamic Graph Latent Variable Model (VDGLVM), a representation learning model based on the variational information bottleneck framework. VDGLVM utilizes a graph generative model and a graph neural network to capture dynamic communication between nodes that one has no access to from the observed data. The proposed computational model provides guaranteed behavior-decoding performance and improves LVMs by associating the inferred latent dynamics with probable DFC.

Learning Task-Oriented Communication for Edge Inference: An Information Bottleneck Approach

This paper investigates task-oriented communication for edge inference, where a low-end edge device transmits the extracted feature vector of a local data sample to a powerful edge server for processing. It is critical to encode the data into an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">informative</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">compact</i> representation for low-latency inference given the limited bandwidth. We propose a learning-based communication scheme that jointly optimizes feature extraction, source coding, and channel coding in a task-oriented manner, i.e., targeting the downstream inference task rather than data reconstruction. Specifically, we leverage an information bottleneck (IB) framework to formalize a rate-distortion tradeoff between the informativeness of the encoded feature and the inference performance. As the IB optimization is computationally prohibitive for the high-dimensional data, we adopt a variational approximation, namely the variational information bottleneck (VIB), to build a tractable upper bound. To reduce the communication overhead, we leverage a sparsity-inducing distribution as the variational prior for the VIB framework to sparsify the encoded feature vector. Furthermore, considering dynamic channel conditions in practical communication systems, we propose a variable-length feature encoding scheme based on dynamic neural networks to adaptively adjust the activated dimensions of the encoded feature to different channel conditions. Extensive experiments evidence that the proposed task-oriented communication system achieves a better rate-distortion tradeoff than baseline methods and significantly reduces the feature transmission latency in dynamic channel conditions.

A Variational Information Bottleneck Method for Network Intrusion Detection

The increasing application of machine learning to network intrusion detection has shown great successes in the recent years. To reduce the high dimension of captured network records for classifiers to learn efficiently, various feature reduction methods have been proposed. Using the Information Bottleneck framework, a concise and discriminative feature representation can be derived by optimizing an intuition and theoretically grounded objective function. In this paper, we propose a novel Variational Information Bottleneck method for Network Intrusion Detection that can produce high-quality low-dimensional representation to increase the classification accuracy of conventional machine learning classifier. Moreover, the proposed model is shown to achieve promising accuracy scores in comparison to several state-of-the-art anomaly-based IDS, on both the 2-class classification and 5-class classification intrusion detection problem.

From Images to Probabilistic Anatomical Shapes: A Deep Variational Bottleneck Approach.

Statistical shape modeling (SSM) directly from 3D medical images is an underutilized tool for detecting pathology, diagnosing disease, and conducting population-level morphology analysis. Deep learning frameworks have increased the feasibility of adopting SSM in medical practice by reducing the expert-driven manual and computational overhead in traditional SSM workflows. However, translating such frameworks to clinical practice requires calibrated uncertainty measures as neural networks can produce over-confident predictions that cannot be trusted in sensitive clinical decision-making. Existing techniques for predicting shape with aleatoric (data-dependent) uncertainty utilize a principal component analysis (PCA) based shape representation computed in isolation of the model training. This constraint restricts the learning task to solely estimating pre-defined shape descriptors from 3D images and imposes a linear relationship between this shape representation and the output (i.e., shape) space. In this paper, we propose a principled framework based on the variational information bottleneck theory to relax these assumptions while predicting probabilistic shapes of anatomy directly from images without supervised encoding of shape descriptors. Here, the latent representation is learned in the context of the learning task, resulting in a more scalable, flexible model that better captures data non-linearity. Additionally, this model is self-regularized and generalizes better given limited training data. Our experiments demonstrate that the proposed method provides an accuracy improvement and better calibrated aleatoric uncertainty estimates than state-of-the-art methods.

GraphComm: Efficient Graph Convolutional Communication for Multiagent Cooperation

Artificial intelligence-empowered smart things (e.g., robots, autonomous vehicles, and unmanned aerial vehicles) have been transforming the world. The “brains” of smart things can be abstracted as the agents or cybertwins residing on end devices and edge servers. The next-generation communication networks (i.e., 6G) will become the nervous system for these agents and natively support multiagent cooperation. By sharing local observations and intentions via communication channels, the agents could better understand the environments and make right decisions. Due to the limited channel bandwidth, the communication is considered as a bottleneck of multiagent cooperation. In this article, we propose a graph convolutional communication method (GraphComm) for multiagent cooperation to relive the bottleneck. Specifically, a variational information bottleneck is used to encode the observations and intentions compactly. Furthermore, a graph information bottleneck with the attention-based neighbor sampling mechanism is utilized to improve the effectiveness and robustness of the multiround communication process. The experimental results show that GraphComm can improve the effectiveness, robustness, and efficiency of communication in multiagent cooperative tasks as compared to baseline methods.

Generalization in Quantum Machine Learning: A Quantum Information Standpoint

Quantum classification and hypothesis testing (state and channel discrimination) are two tightly related subjects, the main difference being that the former is data driven: how to assign to quantum states ρ(x) the corresponding class c (or hypothesis) is learnt from examples during training, where x can be either tunable experimental parameters or classical data “embedded” into quantum states. Does the model generalize? This is the main question in any data-driven strategy, namely the ability to predict the correct class even of previously unseen states. Here we establish a link between quantum classification and quantum information theory, by showing that the accuracy and generalization capability of quantum classifiers depend on the (Rényi) mutual information I(C:Q) and I2(X:Q) between the quantum state space Q and the classical parameter space X or class space C. Based on the above characterization, we then show how different properties of Q affect classification accuracy and generalization, such as the dimension of the Hilbert space, the amount of noise, and the amount of neglected information from X via, e.g., pooling layers. Moreover, we introduce a quantum version of the information bottleneck principle that allows us to explore the various trade-offs between accuracy and generalization. Finally, in order to check our theoretical predictions, we study the classification of the quantum phases of an Ising spin chain, and we propose the variational quantum information bottleneck method to optimize quantum embeddings of classical data to favor generalization.1 MoreReceived 4 March 2021Accepted 30 September 2021DOI:https://doi.org/10.1103/PRXQuantum.2.040321Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.Published by the American Physical SocietyPhysics Subject Headings (PhySH)Research AreasMachine learningQuantum channelsQuantum computationQuantum cryptographyQuantum feedbackQuantum information processingQuantum opticsQuantum sensingQuantum InformationAtomic, Molecular & OpticalStatistical PhysicsInterdisciplinary Physics

Simple and Effective Stochastic Neural Networks

Stochastic neural networks (SNNs) are currently topical, with several paradigms being actively investigated including dropout, Bayesian neural networks, variational information bottleneck (VIB) and noise regularized learning. These neural network variants impact several major considerations, including generalization, network compression, robustness against adversarial attack and label noise, and model calibration. However, many existing networks are complicated and expensive to train, and/or only address one or two of these practical considerations. In this paper we propose a simple and effective stochastic neural network (SE-SNN) architecture for discriminative learning by directly modeling activation uncertainty and encouraging high activation variability. Compared to existing SNNs, our SE-SNN is simpler to implement and faster to train, and produces state of the art results on network compression by pruning, adversarial defense, learning with label noise, and model calibration.

Explaining A Black-box By Using A Deep Variational Information Bottleneck Approach

Interpretable machine learning has gained much attention recently. Briefness and comprehensiveness are necessary in order to provide a large amount of information concisely when explaining a black-box decision system. However, existing interpretable machine learning methods fail to consider briefness and comprehensiveness simultaneously, leading to redundant explanations. We propose the variational information bottleneck for interpretation, VIBI, a system-agnostic interpretable method that provides a brief but comprehensive explanation. VIBI adopts an information theoretic principle, information bottleneck principle, as a criterion for finding such explanations. For each instance, VIBI selects key features that are maximally compressed about an input (briefness), and informative about a decision made by a black-box system on that input (comprehensive). We evaluate VIBI on three datasets and compare with state-of-the-art interpretable machine learning methods in terms of both interpretability and fidelity evaluated by human and quantitative metrics.

Deep visual Re-identification with confidence

Transportation systems often rely on understanding the flow of vehicles or pedestrian. From traffic monitoring at the city scale, to commuters in train terminals, recent progress in sensing technology make it possible to use cameras to better understand the demand, i.e., better track moving agents (e.g., vehicles and pedestrians). Whether the cameras are mounted on drones, vehicles, or fixed in the built environments, they inevitably remain scatter. We need to develop the technology to re-identify the same agents across images captured from non-overlapping field-of-views, referred to as the visual re-identification task. State-of-the-art methods learn a neural network based representation trained with the cross-entropy loss function. We argue that such loss function is not suited for the visual re-identification task hence propose to model confidence in the representation learning framework. We show the impact of our confidence-based learning framework with three methods: label smoothing, confidence penalty, and deep variational information bottleneck. They all show a boost in performance validating our claim. Our contribution is generic to any agent of interest, i.e., vehicles or pedestrians, and outperform highly specialized state-of-the-art methods across 6 datasets. The source code and models are shared towards an open science mission.

Nonlinear quality-related fault detection using combined deep variational information bottleneck and variational autoencoder

Deep learning has gotten much attention in industrial field, many fault detection methods based on deep learning have been developed for nonlinear industrial processes. However, most of them do not take the quality-related faults into account. In order to extract the latent variables which can represent the separated quality-related and unrelated information, this paper proposes a novel deep VIB–VAE algorithm, which combines variational autoencoder (VAE) model and deep variational information bottleneck (VIB). Deep VIB extracts quality-related latent variables by maximizing mutual information between latent variables and process quality while minimizing mutual information between latent variables and observation. VAE is used to learn the quality-unrelated part with above quality-related latent variables as auxiliary information. To monitor and distinguish quality-related and quality-unrelated faults, two monitoring statistics are designed by the two-part latent variables. The reconstruction error by VAE is introduced to improve the performance of fault detection. Finally, the effectiveness of the proposed deep VIB–VAE algorithm is demonstrated by a numerical case and a real hot strip mill process case, respectively.

Novel Convolutional Neural Network with Variational Information Bottleneck for P300 Detection.

In the area of brain-computer interfaces (BCI), the detection of P300 is a very important technique and has a lot of applications. Although this problem has been studied for decades, it is still a tough problem in electroencephalography (EEG) signal processing owing to its high dimension features and low signal-to-noise ratio (SNR). Recently, neural networks, like conventional neural networks (CNN), has shown excellent performance on many applications. However, standard convolutional neural networks suffer from performance degradation on dealing with noisy data or data with too many redundant information. In this paper, we proposed a novel convolutional neural network with variational information bottleneck for P300 detection. Wiht the CNN architecture and information bottleneck, the proposed network termed P300-VIB-Net could remove the redundant information in data effectively. The experimental results on BCI competition data sets show that P300-VIB-Net achieves cutting-edge character recognition performance. Furthermore, the proposed model is capable of restricting the flow of irrelevant information adaptively in the network from perspective of information theory. The experimental results show that P300-VIB-Net is a promising tool for P300 detection.

Learning Extremal Representations with Deep Archetypal Analysis

Archetypes represent extreme manifestations of a population with respect to specific characteristic traits or features. In linear feature space, archetypes approximate the data convex hull allowing all data points to be expressed as convex mixtures of archetypes. As mixing of archetypes is performed directly on the input data, linear Archetypal Analysis requires additivity of the input, which is a strong assumption unlikely to hold e.g. in case of image data. To address this problem, we propose learning an appropriate latent feature space while simultaneously identifying suitable archetypes. We thus introduce a generative formulation of the linear archetype model, parameterized by neural networks. By introducing the distance-dependent archetype loss, the linear archetype model can be integrated into the latent space of a deep variational information bottleneck and an optimal representation, together with the archetypes, can be learned end-to-end. Moreover, the information bottleneck framework allows for a natural incorporation of arbitrarily complex side information during training. As a consequence, learned archetypes become easily interpretable as they derive their meaning directly from the included side information. Applicability of the proposed method is demonstrated by exploring archetypes of female facial expressions while using multi-rater based emotion scores of these expressions as side information. A second application illustrates the exploration of the chemical space of small organic molecules. By using different kinds of side information we demonstrate how identified archetypes, along with their interpretation, largely depend on the side information provided.

The Conditional Entropy Bottleneck.

Much of the field of Machine Learning exhibits a prominent set of failure modes, including vulnerability to adversarial examples, poor out-of-distribution (OoD) detection, miscalibration, and willingness to memorize random labelings of datasets. We characterize these as failures of robust generalization, which extends the traditional measure of generalization as accuracy or related metrics on a held-out set. We hypothesize that these failures to robustly generalize are due to the learning systems retaining too much information about the training data. To test this hypothesis, we propose the Minimum Necessary Information (MNI) criterion for evaluating the quality of a model. In order to train models that perform well with respect to the MNI criterion, we present a new objective function, the Conditional Entropy Bottleneck (CEB), which is closely related to the Information Bottleneck (IB). We experimentally test our hypothesis by comparing the performance of CEB models with deterministic models and Variational Information Bottleneck (VIB) models on a variety of different datasets and robustness challenges. We find strong empirical evidence supporting our hypothesis that MNI models improve on these problems of robust generalization.