Variational Information Bottleneck Research Articles

Adaptive Wireless Image Transmission Transformer Architecture for Image Transmission and Reconstruction.

The advancement of 6G (6th Generation Mobile Networks) communication technology has posed challenges for traditional communication network architectures in meeting the demands for communication efficiency and quality. Semantic communication technology, characterized by its "understand before transmit" approach, has emerged as a pivotal technology driving the progress of 6G due to its ability to enhance communication efficiency and quality. The Wireless Image Transmission Transformer (WITT) model, which operates as a semantic communication system leveraging vision transformer technology for the transmission of semantic images, has shown efficacy in transmitting input images through processes of feature extraction and channel adaptation. This study introduces an advanced channel adaptive module that is informed by deep learning methodologies and the adaptive modulation principles of the Variational Information Bottleneck (VIB). This innovation enhances the original WITT model, resulting in the development of the Adaptive Wireless Image Transmission Transformer (ADWITT) architecture. Comprehensive experimental results have unequivocally shown that the transmission performance of the ADWITT architecture substantially surpasses that of the conventional WITT (Wavelet Image Transmission Technique) model, particularly in scenarios characterized by harsh and detrimental channel conditions. These findings underscore the robustness and adaptability of the ADWITT approach, which is poised to improve the field of image transmission by offering superior performance and resilience in environments where traditional methods falter.

Information filtering and interpolating for semi-supervised graph domain adaptation

Graph domain adaptation, which falls under the umbrella of graph transfer learning, involves transferring knowledge from a labeled source graph to improve prediction accuracy on an unlabeled target graph, where both graphs have identical label spaces but exhibit distribution discrepancies due to temporal data shifts or distinct data collection methods. This adaptation is complicated by the challenges of graph-specific domain discrepancies and cross-graph label scarcity. This paper proposes a semi-supervised Graph domain adaptation method via Information Filtering and Interpolating (GIFI). Specifically, GIFI utilizes a parameterized graph reduction module and a variational information bottleneck to adequately filter out irrelevant information from the source and target graphs to eliminate distribution discrepancy. GIFI also introduces an interpolation-enhanced pseudo-labeling strategy for cross-graph semi-supervised learning, which can mitigate model over-fitting on domain-specific features and limited labeled nodes, thus improving the model’s adaptation and discriminative capability. Experimental results on various graph domain adaptation benchmarks demonstrate GIFI’s superior performance over state-of-the-art methods. Our code is available at https://github.com/joe817/GIFI.

Patch-Level Feature Selection for Thoracic Disease Classification by Chest X-ray Images Using Information Bottleneck.

Chest X-ray (CXR) examination serves as a widely employed clinical test in medical diagnostics. Many studied have tried to apply artificial intelligence (AI) programs to analyze CXR images. Despite numerous positive outcomes, assessing the applicability of AI models for comprehensive diagnostic support remains a formidable challenge. We observed that, even when AI models exhibit high accuracy on one dataset, their performance may deteriorate when tested on another. To address this issue, we propose incorporating a variational information bottleneck (VIB) at the patch level to enhance the generalizability of diagnostic support models. The VIB introduces a probabilistic model aimed at approximating the posterior distribution of latent variables given input data, thereby enhancing the model's generalization capabilities on unseen data. Unlike the conventional VIB approaches that flatten features and use a re-parameterization trick to sample a new latent feature, our method applies the trick to 2D feature maps. This design allows only important pixels to respond, and the model will select important patches in an image. Moreover, the proposed patch-level VIB seamlessly integrates with various convolutional neural networks, offering a versatile solution to improve performance. Experimental results illustrate enhanced accuracy in standard experiment settings. In addition, the method shows robust improvement when training and testing on different datasets.

Explainable Earnings Call Representation Learning (Student Abstract)

Earnings call transcripts hold valuable insights that are vital for investors and analysts when making informed decisions. However, extracting these insights from lengthy and complex transcripts can be a challenging task. The traditional manual examination is not only time-consuming but also prone to errors and biases. Deep learning-based representation learning methods have emerged as promising and automated approaches to tackle this problem. Nevertheless, they may encounter significant challenges, such as the unreliability of the representation encoding process and certain domain-specific requirements in the context of finance. To address these issues, we propose a novel transcript representation learning model. Our model leverages the structural information of transcripts to effectively extract key insights, while endowing model with explainability via variational information bottleneck. Extensive experiments on two downstream financial tasks demonstrate the effectiveness of our approach.

RedCore: Relative Advantage Aware Cross-Modal Representation Learning for Missing Modalities with Imbalanced Missing Rates

Multimodal learning is susceptible to modality missing, which poses a major obstacle for its practical applications and, thus, invigorates increasing research interest. In this paper, we investigate two challenging problems: 1) when modality missing exists in the training data, how to exploit the incomplete samples while guaranteeing that they are properly supervised? 2) when the missing rates of different modalities vary, causing or exacerbating the imbalance among modalities, how to address the imbalance and ensure all modalities are well-trained. To tackle these two challenges, we first introduce the variational information bottleneck (VIB) method for the cross-modal representation learning of missing modalities, which capitalizes on the available modalities and the labels as supervision. Then, accounting for the imbalanced missing rates, we define relative advantage to quantify the advantage of each modality over others. Accordingly, a bi-level optimization problem is formulated to adaptively regulate the supervision of all modalities during training. As a whole, the proposed approach features Relative advantage aware Cross-modal representation learning (abbreviated as RedCore) for missing modalities with imbalanced missing rates. Extensive empirical results demonstrate that RedCore outperforms competing models in that it exhibits superior robustness against either large or imbalanced missing rates. The code is available at: https://github.com/sunjunaimer/RedCore.

MVIB-DVA: Learning minimum sufficient multi-feature speech emotion embeddings under dual-view aware

Speech emotion recognition (SER) is a crucial topic in human–computer interaction. However, there are still many challenges to extracting emotional embeddings. Emotional embeddings extracted by network models often contain noise and incomplete emotional information. To meet these challenges, this study developed an innovative model (MVIB-DVA) composed of a multi-feature variational information bottleneck (MVIB) based on the information bottleneck (IB) principle and a dual-view aware module (DVAM) with an attention mechanism. MVIB employs the IB principle as the driving model and utilizes learned minimal sufficient single-feature emotional embeddings as auxiliary information. The aims are to capture unique emotional information in individual features and complementary information between different types of features as well as reduce noise and represent rich emotional information with fewer parameters. DVAM proposes (1) a frequency-domain statistical aware module (FDSAM) in the frequency domain that emphasizes the frequency that best reflects emotional information and (2) a frame aware module (FAM) in the time domain that focuses on the frames that contribute to the most to the final emotion recognition results. A separate channel extracts details ignored in the frequency and time domain views, extracting more comprehensive emotional information. The experimental results show that our method performs excellently in recognizing speech emotions. MVIB-DVA achieved weighted accuracy (WA) of 74.05% and unweighted accuracy (UA) of 75.67% on the IEMOCAP dataset. Similarly, on the RAVDESS dataset, MVIB-DVA attained WA of 86.66% and UA of 86.51%.

EVAE: Evolutionary Variational Autoencoder.

Variational autoencoders (VAEs) are challenged by the imbalance between representation inference and task fitting caused by surrogate loss. To address this issue, existing methods adjust their balance by directly tuning their coefficients. However, these methods suffer from a tradeoff uncertainty, i.e., nondynamic regulation over iterations and inflexible hyperparameters for learning tasks. Accordingly, we make the first attempt to introduce an evolutionary VAE (eVAE), building on the variational information bottleneck (VIB) theory and integrative evolutionary neural learning. eVAE integrates a variational genetic algorithm (VGA) into VAE with variational evolutionary operators, including variational mutation (V-mutation), crossover, and evolution. Its training mechanism synergistically and dynamically addresses and updates the learning tradeoff uncertainty in the evidence lower bound (ELBO) without additional constraints and hyperparameter tuning. Furthermore, eVAE presents an evolutionary paradigm to tune critical factors of VAEs and addresses the premature convergence and random search problem in integrating evolutionary optimization into deep learning. Experiments show that eVAE addresses the KL-vanishing problem for text generation with low reconstruction loss, generates all the disentangled factors with sharp images, and improves image generation quality. eVAE achieves better disentanglement, generation performance, and generation-inference balance than its competitors. Code available at: https://github.com/amasawa/eVAE.

Synergetic Decomposition of Input-output Dependency for Control System Intelligent Monitoring: A Perspective from Information Theory

Control systems regulate industrial processes by processing input signals to generate corresponding output signals. The monitoring of control systems ensures process safety. For control systems, we have the following recognition: the future outputs are determined by the output states and input excitations at the past moments, which have different interpretability capacities. It is a significant challenge to decouple the intricate relationship between inputs and outputs to provide comprehensive and interpretable monitoring results for control systems. In this paper, we propose a synergetic decomposition variational information bottleneck (SDVIB) approach to solve this problem from the perspective of information theory. Firstly, a compressed representation strategy is proposed to extract interpretable features from the output and input separately by employing the information bottleneck principle, which can provide initial state information and excitation information, respectively. Secondly, a reconstruction model is established within the framework of variational autoencoder (VAE), which can decouple the remaining interpretable-unrelated features to describe the cross-correlation separately among the input and output variables. In this way, the relationships between input and output variables are decoupled into multiple subspaces with specific physical meanings, which can be described by the constructed monitoring statistics. The effectiveness of the proposed method is validated through a well-known benchmark process.

Adversarially robust neural networks with feature uncertainty learning and label embedding

Deep neural networks (DNNs) are vulnerable to the attacks of adversarial examples, which bring serious security risks to the learning systems. In this paper, we propose a new defense method to improve the adversarial robustness of DNNs based on stochastic neural networks (SNNs), termed as Margin-SNN. The proposed Margin-SNN mainly includes two modules, i.e., feature uncertainty learning module and label embedding module. The first module introduces uncertainty to the latent feature space by giving each sample a distributional representation rather than a fixed point representation, and leverages the advantages of variational information bottleneck method in achieving good intra-class compactness in latent space. The second module develops a label embedding mechanism to take advantage of the semantic information underlying the labels, which maps the labels into the same latent space with the features, in order to capture the similarity between sample and its class centroid, where a penalty term is equipped to elegantly enlarge the margin between different classes for better inter-class separability. Since no adversarial information is introduced, the proposed model can be learned in standard training to improve adversarial robustness, which is much more efficient than adversarial training. Extensive experiments on data sets MNIST, FASHION MNIST, CIFAR10, CIFAR100 and SVHN demonstrate superior defensive ability of the proposed method. Our code is available at https://github.com/humeng24/Margin-SNN.

Contrastive variational information bottleneck for aspect-based sentiment analysis

Deep learning techniques have dominated the literature on aspect-based sentiment analysis (ABSA), achieving state-of-the-art performance. However, deep models generally suffer from spurious correlations between input features and output labels, which significantly hurts the robustness and generalization capability. In this paper, we propose to reduce spurious correlations for ABSA, via a novel Contrastive Variational Information Bottleneck framework (called CVIB). The proposed CVIB framework is composed of an original network and a self-pruned network, and these two networks are optimized simultaneously via contrastive learning. Concretely, we employ the Variational Information Bottleneck (VIB) principle to learn an informative and compressed network (self-pruned network) from the original network, which discards the superfluous patterns or spurious correlations between input features and prediction labels. Then, self-pruning contrastive learning is devised to pull together semantically similar positive pairs and push away dissimilar pairs, where the representations of the anchor learned by the original and self-pruned networks respectively are regarded as a positive pair while the representations of two different sentences within a mini-batch are treated as a negative pair. To verify the effectiveness of our CVIB method, we conduct extensive experiments on five benchmark ABSA datasets. The experimental results show that our approach achieves better performance than the strong competitors in terms of overall prediction performance, robustness, and generalization.

Privacy‐preserving graph publishing with disentangled variational information bottleneck

SummarySocial networks collect enormous amounts of user personal and behavioral data, which could threaten users' privacy if published or shared directly. Privacy‐preserving graph publishing (PPGP) can make user data available while protecting private information. For this purpose, in PPGP, anonymization methods like perturbation and generalization are commonly used. However, traditional anonymization methods are challenging in balancing high‐level privacy and utility, ineffective at defending against both various link and hybrid inference attacks, as well as vulnerable to graph neural network (GNN)‐based attacks. To solve those problems, we present a novel privacy‐disentangled approach that disentangles private and non‐private information for a better privacy‐utility trade‐off. Moreover, we propose a unified graph deep learning framework for PPGP, denoted privacy‐disentangled variational information bottleneck (PDVIB). Using low‐dimensional perturbations, the model generates an anonymized graph to defend against various inference attacks, including GNN‐based attacks. Particularly, the model fits various privacy settings by employing adjustable perturbations at the node level. With three real‐world datasets, PDVIB is demonstrated to generate robust anonymous graphs that defend against various privacy inference attacks while maintaining the utility of non‐private information.

MV-CVIB: a microbiome-based multi-view convolutional variational information bottleneck for predicting metastatic colorectal cancer.

Imbalances in gut microbes have been implied in many human diseases, including colorectal cancer (CRC), inflammatory bowel disease, type 2 diabetes, obesity, autism, and Alzheimer's disease. Compared with other human diseases, CRC is a gastrointestinal malignancy with high mortality and a high probability of metastasis. However, current studies mainly focus on the prediction of colorectal cancer while neglecting the more serious malignancy of metastatic colorectal cancer (mCRC). In addition, high dimensionality and small samples lead to the complexity of gut microbial data, which increases the difficulty of traditional machine learning models. To address these challenges, we collected and processed 16S rRNA data and calculated abundance data from patients with non-metastatic colorectal cancer (non-mCRC) and mCRC. Different from the traditional health-disease classification strategy, we adopted a novel disease-disease classification strategy and proposed a microbiome-based multi-view convolutional variational information bottleneck (MV-CVIB). The experimental results show that MV-CVIB can effectively predict mCRC. This model can achieve AUC values above 0.9 compared to other state-of-the-art models. Not only that, MV-CVIB also achieved satisfactory predictive performance on multiple published CRC gut microbiome datasets. Finally, multiple gut microbiota analyses were used to elucidate communities and differences between mCRC and non-mCRC, and the metastatic properties of CRC were assessed by patient age and microbiota expression.

Rethinking Feature Generalization in Vacant Space Detection.

Vacant space detection is critical in modern parking lots. However, deploying a detection model as a service is not an easy task. As the camera in a new parking is set up at different heights or viewing angles from the original parking lot where the training data are collected, the performance of the vacant space detector could be degraded. Therefore, in this paper, we proposed a method to learn generalized features so that the detector can work better in different environments. In detail, the features are suitable for a vacant detection task and robust to environmental change. We use a reparameterization process to model the variance from the environment. In addition, a variational information bottleneck is used to ensure the learned feature focus on only the appearance of a car in a specific parking space. Experimental results show that performances on a new parking lot increase significantly when only data from source parking are used in the training phase.

Spaceborne Cooperative Detection for Distributed Sensing: Overcoming Inter-Satellite Link Limitations via Deep Information Bottleneck

We consider the cooperative detection for distributed sensing via low Earth orbit constellations. The source data gathered by the ground sensors are non-orthogonally sent to satellite access points, and then aggregated at a central satellite via inter-satellite links (ISLs). To overcome the intrinsic ISL bandwidth limitation, we consider the deep auto-encoding paradigm to jointly design the ISL transceivers among satellites, and propose a novel deep variational information bottleneck (DVIB) method which maximizes the end-to-end sensing accuracy under bandwidth constraints. Specifically, the mathematically untractable ISL bandwidth constraint is first transformed into an entropy-based format. Then a customized batch-norm layer is introduced, where the messages on ISLs are considered as latent variables and are regularized with entropy-constrained posterior for efficient compression. Compared to the benchmark, the proposed DVIB method is shown to simultaneously reduce the bandwidth overhead by 30% and enhance the sensing accuracy by 2–5 dB, validating the significance of relevant information extraction on ISLs.

Variable Selection Using Deep Variational Information Bottleneck with Drop-Out-One Loss

The information bottleneck (IB) model aims to find the optimal representations of input variables with respect to the response variable. While it has been widely used in the machine-learning community, research from the perspective of the information-theoretic method has been rarely reported regarding variable selection. In this paper, we investigate DNNs for variable selection through an information-theoretic lens. To be specific, we first state the rationality of variable selection with IB and then propose a new statistic to measure the variable importance. On this basis, a new algorithm based on a deep variational information bottleneck is developed to calculate the statistic, in which we consider the Gaussian distribution and the exponential distribution to estimate the Kullback–Leibler divergence. Empirical evaluations on simulated and real-world data show that the proposed method performs better than classical variable-selection methods. This confirms the feasibility of the variable selection from the perspective of IB.

Variational Information Bottleneck Regularized Deep Reinforcement Learning for Efficient Robotic Skill Adaptation

Deep Reinforcement Learning (DRL) algorithms have been widely studied for sequential decision-making problems, and substantial progress has been achieved, especially in autonomous robotic skill learning. However, it is always difficult to deploy DRL methods in practical safety-critical robot systems, since the training and deployment environment gap always exists, and this issue would become increasingly crucial due to the ever-changing environment. Aiming at efficiently robotic skill transferring in a dynamic environment, we present a meta-reinforcement learning algorithm based on a variational information bottleneck. More specifically, during the meta-training stage, the variational information bottleneck first has been applied to infer the complete basic tasks for the whole task space, then the maximum entropy regularized reinforcement learning framework has been used to learn the basic skills consistent with that of basic tasks. Once the training stage is completed, all of the tasks in the task space can be obtained by a nonlinear combination of the basic tasks, thus, the according skills to accomplish the tasks can also be obtained by some way of a combination of the basic skills. Empirical results on several highly nonlinear, high-dimensional robotic locomotion tasks show that the proposed variational information bottleneck regularized deep reinforcement learning algorithm can improve sample efficiency by 200-5000 times on new tasks. Furthermore, the proposed algorithm achieves substantial asymptotic performance improvement. The results indicate that the proposed meta-reinforcement learning framework makes a significant step forward to deploy the DRL-based algorithm to practical robot systems.

Self-Supervised Learning for High-Resolution Remote Sensing Images Change Detection With Variational Information Bottleneck

Self-Supervised Learning for High-Resolution Remote Sensing Images Change Detection With Variational Information Bottleneck

Fully Bayesian VIB-DeepSSM.

Statistical shape modeling (SSM) enables population-based quantitative analysis of anatomical shapes, informing clinical diagnosis. Deep learning approaches predict correspondence-based SSM directly from unsegmented 3D images but require calibrated uncertainty quantification, motivating Bayesian formulations. Variational information bottleneck DeepSSM (VIB-DeepSSM) is an effective, principled framework for predicting probabilistic shapes of anatomy from images with aleatoric uncertainty quantification. However, VIB is only half-Bayesian and lacks epistemic uncertainty inference. We derive a fully Bayesian VIB formulation and demonstrate the efficacy of two scalable implementation approaches: concrete dropout and batch ensemble. Additionally, we introduce a novel combination of the two that further enhances uncertainty calibration via multimodal marginalization. Experiments on synthetic shapes and left atrium data demonstrate that the fully Bayesian VIB network predicts SSM from images with improved uncertainty reasoning without sacrificing accuracy.

Progressive Interpretation Synthesis: Interpreting Task Solving by Quantifying Previously Used and Unused Information.

A deep neural network is a good task solver, but it is difficult to make sense of its operation. People have different ideas about how to interpret its operation. We look at this problem from a new perspective where the interpretation of task solving is synthesized by quantifying how much and what previously unused information is exploited in addition to the information used to solve previous tasks. First, after learning several tasks, the network acquires several information partitions related to each task. We propose that the network then learns the minimal information partition that supplements previously learned information partitions to more accurately represent the input. This extra partition is associated with unconceptualized information that has not been used in previous tasks. We manage to identify what unconceptualized information is used and quantify the amount. To interpret how the network solves a new task, we quantify as meta-information how much information from each partition is extracted. We implement this framework with the variational information bottleneck technique. We test the framework with the MNIST and the CLEVR data set. The framework is shown to be able to compose information partitions and synthesize experience-dependent interpretation in the form of meta-information. This system progressively improves the resolution of interpretation upon new experience by converting a part of the unconceptualized information partition to a task-related partition. It can also provide a visual interpretation by imaging what is the part of previously unconceptualized information that is needed to solve a new task.

Attentive Variational Information Bottleneck for TCR-peptide Interaction Prediction.

We present a multi-sequence generalization of Variational Information Bottleneck (VIB) (Alemi et al., 2016) and call the resulting model Attentive Variational Information Bottleneck (AVIB). Our AVIB model leverages multi-head self-attention (Vaswani et al., 2017) to implicitly approximate a posterior distribution over latent encodings conditioned on multiple input sequences. We apply AVIB to a fundamental immuno-oncology problem: predicting the interactions between T cell receptors (TCRs) and peptides. Experimental results on various datasets show that AVIB significantly outperforms state-of-the-art methods for TCR-peptide interaction prediction. Additionally, we show that the latent posterior distribution learned by AVIB is particularly effective for the unsupervised detection of out-of-distribution (OOD) amino acid sequences. The code and the data used for this study are publicly available at: https://github.com/nec-research/vibtcr. Supplementary data are available at Bioinformatics online.