Mutual Information Estimation Research Articles

Semantic-Driven Crossmodal Fusion for Multimodal Sentiment Analysis

In multimodal sentiment analysis (MSA), the fusion strategies of multimodal features significantly influence the performance of MSA models. Previous works frequently face challenges in integrating heterogeneous data without fully leveraging the rich semantic content of text, resulting in poor information association. We propose an MSA model based on Text-Driven Crossmodal Fusion and Mutual Information Estimation, called TeD-MI, which comprises a Stacked Text-Driven Crossmodal Fusion (STDC) module, which efficiently fusions the three modalities driven by the text modality to optimize fusion feature representation and enhance semantic understanding. Furthermore, TeD-MI designed a mutual information estimation module to achieve the best balance between preserving task-related information and filtering out irrelevant noise information as much as possible. Comprehensive experiments conducted on the CMU-MOSI and CMU-MOSEI datasets demonstrate our proposed model achieves varying degrees of improvement across most evaluation metrics.

Heterogeneous graph representation learning via mutual information estimation for fraud detection

In the fraud detection, fraudsters frequently engage with numerous benign users to disguise their activities. Consequently, the fraud graph exhibits not only homogeneous connections between the fraudsters and the same labelled nodes, but also heterogeneous connections, where fraudsters interact with the legitimate nodes. Heterogeneous graph representation learning aims at extracting the structural and semantic information and embed it into the low-dimensional node representation. Recently, maximizing the mutual information between the local node embedding and the summary representation has achieved the promising results on node classification tasks. However, existing deep graph infomax methods still have the following limitations. Firstly, attribute information of nodes in the graph is not fully utilized for capturing the semantic relationships between nodes. Secondly, the local and global supervision signal are not simultaneously exploited for the node embedding learning. Thirdly, the multiplex heterogeneous relations among nodes are ignored. To address these issues, a heterogeneous graph representation learning model by mutual information estimation (MIE-HetGRL) is proposed in this paper to identify the fraudsters in the fraud review graph. Concretely, a high-order mutual information estimation is proposed to integrate the local and global mutual information as the supervision signal. Then we devise a semantic attention fusion module to aggregate the relation-aware node embeddings into a compact node representation. Finally, a joint contrastive learning is designed for facilitating the training and optimization of model. The experimental results show that our proposed model significantly outperforms state-of-the-art baselines for fraud detection.

Learning personalized drug features and differentiated drug-pair interaction information for drug–drug interaction prediction

Multi-drug combination therapies are increasingly used for complex diseases but carry risks of harmful drug interactions. Effective drug–drug interaction prediction (DDIP) is essential for assessing risks among numerous drug pairs. Most DDIP methods involve two main steps: drug representation and drug pair interaction extraction, respectively challenged by the loss of personalized drug information and the need for differentiated interaction data, and are rarely studied. Specifically, personalized drug information refers to the distinct features of each drug. These properties can be easily confused by neighboring information during graph propagation. This issue is especially prominent in drug interaction graphs with long-tail distributions, which poses challenges for personalized drug information learning. Furthermore, it is crucial to learn interactions with differentiation in order to identify diverse drug relationships. Some methods simply concatenate drug features, often ignoring the differences of different drug relationships, while other methods based on substructures rely on professional pharmacological knowledge and are computationally complex. To address these issues, we propose a novel method, learning personalized Drug Features and differentiated Drug-Pair interaction information for drug–drug interaction prediction (DFPDDI). This approach employs a contrastive learning network with edge-aware augmentations and mutual information estimators to capture personalized drug features across various graph distributions. Furthermore, it applies a mutual information constraint to drug-pair representations, enhancing the accuracy of interaction predictions by better distinguishing between different types of drug relationships. The results evaluated on three public datasets demonstrate competitive performance compared to baselines. It also shows potential for accurate predictions, particularly in imbalanced-distribution graphs.

A comparative analysis of mutual information methods for pairwise relationship detection in metagenomic data

BackgroundConstruction of co-occurrence networks in metagenomic data often employs correlation to infer pairwise relationships between microbes. However, biological systems are complex and often display qualities non-linear in nature. Therefore, the reliance on correlation alone may overlook important relationships and fail to capture the full breadth of intricacies presented in underlying interaction networks. It is of interest to incorporate metrics that are not only robust in detecting linear relationships, but non-linear ones as well.ResultsIn this paper, we explore the use of various mutual information (MI) estimation approaches for quantifying pairwise relationships in biological data and compare their performances against two traditional measures–Pearson’s correlation coefficient, r, and Spearman’s rank correlation coefficient, ρ. Metrics are tested on both simulated data designed to mimic pairwise relationships that may be found in ecological systems and real data from a previous study on C. diff infection. The results demonstrate that, in the case of asymmetric relationships, mutual information estimators can provide better detection ability than Pearson’s or Spearman’s correlation coefficients. Specifically, we find that these estimators have elevated performances in the detection of exploitative relationships, demonstrating the potential benefit of including them in future metagenomic studies.ConclusionsMutual information (MI) can uncover complex pairwise relationships in biological data that may be missed by traditional measures of association. The inclusion of such relationships when constructing co-occurrence networks can result in a more comprehensive analysis than the use of correlation alone.

Multi-label feature selection based on minimizing feature redundancy of mutual information

Multi-label feature selection is an indispensable technology in the preprocessing of multi-label high-dimensional data. Approaches utilizing information theory and sparse models hold promise in this domain, demonstrating strong performance. Although there have been extensive literatures using l1 and l2,1-norms to identify label-specific features and common features in the feature space, they all ignore the redundant information interference problem when different features are learned simultaneously. Considering that features and labels in multi-label data are rarely linearly correlated, the MFS-MFR approach is presented to generate a representation of the nonlinear correlation between features and labels using the mutual information estimator. Following that, MFS-MFR detects specific and common features in the feature-label mutual information space using two coefficient matrices constrained by the l1 and l2,1-norms, respectively. In particular, we define a non-zero correlation constraint that effectively minimizes the redundant correlation between the two matrices. Moreover, a manifold regularization term is devised to preserve the local information of the mutual information space. To solve the optimization model with nonlinear binary regular term, we employ a novel solution approach called S-FISTA. Extensive experiments across 15 multi-label benchmark datasets, comparing against 11 top-performing multi-label feature selection methods, demonstrate the superior performance of MFS-MFR.

Augmentation-Free Graph Contrastive Learning of Invariant-Discriminative Representations.

Graph contrastive learning (GCL) is a promising direction toward alleviating the label dependence, poor generalization and weak robustness of graph neural networks, learning representations with invariance, and discriminability by solving pretasks. The pretasks are mainly built on mutual information estimation, which requires data augmentation to construct positive samples with similar semantics to learn invariant signals and negative samples with dissimilar semantics to empower representation discriminability. However, an appropriate data augmentation configuration depends heavily on lots of empirical trials such as choosing the compositions of data augmentation techniques and the corresponding hyperparameter settings. We propose an augmentation-free GCL method, invariant-discriminative GCL (iGCL), that does not intrinsically require negative samples. iGCL designs the invariant-discriminative loss (ID loss) to learn invariant and discriminative representations. On the one hand, ID loss learns invariant signals by directly minimizing the mean square error (MSE) between the target samples and positive samples in the representation space. On the other hand, ID loss ensures that the representations are discriminative by an orthonormal constraint forcing the different dimensions of representations to be independent of each other. This prevents representations from collapsing to a point or subspace. Our theoretical analysis explains the effectiveness of ID loss from the perspectives of the redundancy reduction criterion, canonical correlation analysis (CCA), and information bottleneck (IB) principle. The experimental results demonstrate that iGCL outperforms all baselines on five node classification benchmark datasets. iGCL also shows superior performance for different label ratios and is capable of resisting graph attacks, which indicates that iGCL has excellent generalization and robustness. The source code is available at https://github.com/lehaifeng/ T-GCN/tree/master/iGCL.

Enhanced mutual information neural estimators for optical fiber communication.

The accurate estimation of mutual information (MI) plays a vital role in understanding channel capacity and optimizing the performance of optical communications. While MI computations for the additive white Gaussian noise (AWGN) channel are well-established, they fall short when dealing with the challenges posed by nonlinear optical fiber channels due to an unknown channel model. For the first time, to our knowledge, this Letter introduces a mutual information neural estimator (MINE) for MI estimation in optical fiber communications. We propose an enhanced MINE (E-MINE), achieved by enlarging the training batch size to improve estimation accuracy and stability. Our findings reveal that the E-MINE achieves highly accurate estimations in the AWGN channel and maintains strong consistency with symbol-by-symbol MI estimations, comparable to Monte Carlo (MC) methods based on a Gaussian distribution in long-haul optical fiber channels. Furthermore, with multi-symbol estimation, the E-MINE yields a 0.16 bits/4D-symbol improvement in our experiments. We anticipate that our findings will drive further research in the field, opening new possibilities for enhancing communication systems design and performance using deep learning techniques.

Artifact feature purification for cross-domain detection of AI-generated images

In the era of AIGC, the fast development of visual content generation technologies, such as diffusion models, brings potential security risks to our society. Existing generated image detection methods suffer from performance drops when faced with out-of-domain generators and image scenes. To relieve this problem, we propose Artifact Purification Network (APN) to facilitate the artifact extraction from generated images through the explicit and implicit purification processes. For the explicit one, a suspicious frequency-band proposal method and a spatial feature decomposition method are proposed to extract artifact-related features. For the implicit one, a training strategy based on mutual information estimation is proposed to further purify the artifact-related features. The experiments are conducted in two settings. Firstly, we perform a cross-generator evaluation, wherein detectors trained using data from one generator are evaluated on data generated by other generators. Secondly, we conduct a cross-scene evaluation, wherein detectors trained for a specific domain of content (e.g., ImageNet) are assessed on data collected from another domain (e.g., LSUN-Bedroom). Results show that for cross-generator detection, the average accuracy of APN is 5.6%∼16.4% higher than the previous 11 methods on the GenImage dataset and 1.7%∼50.1% on the DiffusionForensics dataset. For cross-scene detection, APN maintains its high performance. Via visualization analysis, we find that the proposed method can extract diverse forgery patterns and condense the forgery information diluted in irrelated features. We also find that the artifact features APN focuses on across generators and scenes are global and diverse. The code will be available at https://github.com/RichardSunnyMeng/APN-official-codes.

Estimation of complete mutual information exploiting nonlinear magnitude-phase dependence: Application to spatial FNC for complex-valued fMRI data

BackgroundReal-valued mutual information (MI) has been used in spatial functional network connectivity (FNC) to measure high-order and nonlinear dependence between spatial maps extracted from magnitude-only functional magnetic resonance imaging (fMRI). However, real-valued MI cannot fully capture the group differences in spatial FNC from complex-valued fMRI data with magnitude and phase dependence. MethodsWe propose a complete complex-valued MI method according to the chain rule of MI. We fully exploit the dependence among magnitudes and phases of two complex-valued signals using second and fourth-order joint entropies, and propose to use a Gaussian copula transformation with a lower bound property to avoid inaccurate estimation of joint probability density function when computing the joint entropies. ResultsThe proposed method achieves more accurate MI estimates than the two histogram-based (normal and symbolic approaches) and kernel density estimation methods for simulated signals, and enhances group differences in spatial functional network connectivity for experimental complex-valued fMRI data. Comparison with existing methodsCompared with the simplified complex-valued MI and real-valued MI, the proposed method yields higher MI estimation accuracy, leading to 17.4 % and 145.5 % wider MI ranges, and more significant connectivity differences between healthy controls and schizophrenia patients. A unique connection between executive control network (EC) and right frontal parietal areas, and three additional connections mainly related to EC are detected than the simplified complex-valued MI. ConclusionsWith capability in quantifying MI fully and accurately, the proposed complex-valued MI is promising in providing qualified FNC biomarkers for identifying mental disorders such as schizophrenia.

Robust phase unwrapping algorithm for gear interferometry based on mutual information and adaptive spin-inpainting approaches

An enhanced robust phase unwrapping algorithm for gear tooth flank measurement by laser interferometry is proposed. The method integrates wrapped phase mutual information estimation with an adaptive spin-inpainting strategy, resulting in improved quality phase maps to guide the phase unwrapping process. Mutual information estimation is employed to investigate potential phase relationships among interference fringes in the wrapped phase map, forming the foundation for the suppression and recovery of abnormal fringes. The adaptive spin-inpainting method, which adjusts in response to fringe density, plays a crucial role in efficiently addressing the restoration of abnormal fringes and obtaining the optimized quality map. With the enhanced new quality map, the approach yields optimized results, enabling a concurrent assessment of unwrapping process efficiency and quality. Several simulations and experiments were conducted to validate the effectiveness of the proposed method. Comparative analyses against commonly employed methods consistently demonstrate the superior performance of the proposed algorithm.

Informed Adaptive Sensing.

For many inverse problems, the data on which the solution is based is acquired sequentially. We present an approach to the solution of such inverse problems where a sensor can be directed (or otherwise reconfigured on the fly) to acquire a particular measurement. An example problem is magnetic resonance image reconstruction. We use an estimate of mutual information derived from an empirical conditional distribution provided by a generative model to guide our measurement acquisition given measurements acquired so far. The conditionally generated data is a set of samples which are representative of the plausible solutions that satisfy the acquired measurements. We present experiments on toy and real world data sets. We focus on image data but we demonstrate that the method is applicable to a broader class of problems. We also show how a learned model such as a deep neural network can be leveraged to allow generalisation to unseen data. Our informed adaptive sensing method outperforms random sampling, variance based sampling, sparsity based methods, and compressed sensing.

Neural Joint Entropy Estimation.

Estimating the entropy of a discrete random variable is a fundamental problem in information theory and related fields. This problem has many applications in various domains, including machine learning, statistics, and data compression. Over the years, a variety of estimation schemes have been suggested. However, despite significant progress, most methods still struggle when the sample is small, compared to the variable's alphabet size. In this work, we introduce a practical solution to this problem, which extends the work of McAllester and Statos. The proposed scheme uses the generalization abilities of cross-entropy estimation in deep neural networks (DNNs) to introduce improved entropy estimation accuracy. Furthermore, we introduce a family of estimators for related information-theoretic measures, such as conditional entropy and mutual information (MI). We show that these estimators are strongly consistent and demonstrate their performance in a variety of use cases. First, we consider large alphabet entropy estimation. Then, we extend the scope to MI estimation. Next, we apply the proposed scheme to conditional MI estimation, as we focus on independence testing tasks. Finally, we study a transfer entropy (TE) estimation problem. The proposed estimators demonstrate improved performance compared to existing methods in all of these setups.

A Twist for Graph Classification: Optimizing Causal Information Flow in Graph Neural Networks

Graph neural networks (GNNs) have achieved state-of-the-art results on many graph representation learning tasks by exploiting statistical correlations. However, numerous observations have shown that such correlations may not reflect the true causal mechanisms underlying the data and thus may hamper the ability of the model to generalize beyond the observed distribution. To address this problem, we propose an Information-based Causal Learning (ICL) framework that combines information theory and causality to analyze and improve graph representation learning to transform information relevance to causal dependence. Specifically, we first introduce a multi-objective mutual information optimization objective derived from information-theoretic analysis and causal learning principles to simultaneously extract invariant and interpretable causal information and reduce reliance on non-causal information in correlations. To optimize this multi-objective objective, we enable a causal disentanglement layer that effectively decouples the causal and non-causal information in the graph representations. Moreover, due to the intractability of mutual information estimation, we derive variational bounds that enable us to transform the above objective into a tractable loss function. To balance the multiple information objectives and avoid optimization conflicts, we leverage multi-objective gradient descent to achieve a stable and efficient transformation from informational correlation to causal dependency. Our approach provides important insights into modulating the information flow in GNNs to enhance their reliability and generalization. Extensive experiments demonstrate that our approach significantly improves the robustness and interpretability of GNNs across different distribution shifts. Visual analysis demonstrates how our method converts informative dependencies in representations into causal dependencies.

Hierarchical bottleneck for heterogeneous graph representation

Heterogeneous graphs (HGs) contain many nodes and their interaction relationships, which can model complex systems and provide rich semantic and structural information for task execution. Among these, HG representation stands as the fundamental and pivotal component. Existing HG representation methods primarily employ graph neural networks to acquire the semantics of nodes along various meta-paths and fuse them to represent the nodes. The most prevalent HG representation methods encompass two steps: semantic information extraction within meta-paths and semantic fusion between meta-paths. However, these methods overlooked the consideration of node heterogeneity within meta-paths and the simultaneous semantic correlation between meta-paths. Specifically, node heterogeneity within meta-paths signifies that the meta-path-based neighbors do not consistently contain information that positively influences the target node, and the semantic correlation between meta-paths indicates that different meta-path spaces are not entirely independent. Disregarding either of these issues leads to the propagation of irrelevant or redundant information and potential disruption of HG embedding. Consequently, in this study, we propose the HBHG, which is a hierarchical bottleneck for heterogeneous graph representation. HBHG primarily employs the information bottleneck (IB) as a guiding principle, constraining the propagation of irrelevant information within and between meta-paths while preserving relevant information. The central concept of the IB revolves around viewing model learning as the preservation of relevant information and compression of irrelevant information, accomplished by minimizing the dependency between input and hidden features through mutual information (MI) and maximizing the dependency between hidden features and ground-truth. Considering the complexity associated with MI estimation, this paper introduces a novel dependency index, namely the Hilbert-Schmidt independence criterion (HSIC), which offers ease of calculation. Specifically, HBHG comprises two primary components: a semantic bottleneck within meta-paths and a semantic bottleneck between meta-paths. The semantic bottleneck within meta-paths relies primarily on the HSIC-based limitations of dependencies at different layers of the graph neural network on various meta-paths, thereby maximizing the extraction of information relevant to the target node from neighboring nodes. The semantic bottleneck between meta-paths enables flexible extraction and fusion of semantic information based on downstream tasks, achieved by managing the trade-off of dependencies with HSIC between different meta-path semantic spaces. In summary, the proposed HBHG integrates hierarchical bottleneck constraints within and between meta-paths. This integration serves to maximize the aggregation of relevant information while effectively compressing irrelevant information, thereby enhancing the quality of heterogeneous graph embedding. The effectiveness of HBHG was validated through performance and ablation experiments conducted on multiple datasets.

Slimming Neural Networks Using Adaptive Connectivity Scores.

In general, deep neural network (DNN) pruning methods fall into two categories: 1) weight-based deterministic constraints and 2) probabilistic frameworks. While each approach has its merits and limitations, there are a set of common practical issues such as trial-and-error to analyze sensitivity and hyper-parameters to prune DNNs, which plague them both. In this work, we propose a new single-shot, fully automated pruning algorithm called slimming neural networks using adaptive connectivity scores (SNACS). Our proposed approach combines a probabilistic pruning framework with constraints on the underlying weight matrices, via a novel connectivity measure, at multiple levels to capitalize on the strengths of both approaches while solving their deficiencies. In SNACS, we propose a fast hash-based estimator of adaptive conditional mutual information (ACMI), that uses a weight-based scaling criterion, to evaluate the connectivity between filters and prune unimportant ones. To automatically determine the limit up to which a layer can be pruned, we propose a set of operating constraints that jointly define the upper pruning percentage limits across all the layers in a deep network. Finally, we define a novel sensitivity criterion for filters that measures the strength of their contributions to the succeeding layer and highlights critical filters that need to be completely protected from pruning. Through our experimental validation, we show that SNACS is faster by over 17× the nearest comparable method and is the state-of-the-art single-shot pruning method across four standard Dataset-DNN pruning benchmarks: CIFAR10-VGG16, CIFAR10-ResNet56, CIFAR10-MobileNetv2, and ILSVRC2012-ResNet50.

Learning Channel Capacity With Neural Mutual Information Estimator Based on Message Importance Measure

Learning Channel Capacity With Neural Mutual Information Estimator Based on Message Importance Measure

Recognizing Predictive Substructures With Subgraph Information Bottleneck.

The emergence of Graph Convolutional Network (GCN) has greatly boosted the progress of graph learning. However, two disturbing factors, noise and redundancy in graph data, and lack of interpretation for prediction results, impede further development of GCN. One solution is to recognize a predictive yet compressed subgraph to get rid of the noise and redundancy and obtain the interpretable part of the graph. This setting of subgraph is similar to the information bottleneck (IB) principle, which is less studied on graph-structured data and GCN. Inspired by the IB principle, we propose a novel subgraph information bottleneck (SIB) framework to recognize such subgraphs, named IB-subgraph. However, the intractability of mutual information and the discrete nature of graph data makes the objective of SIB notoriously hard to optimize. To this end, we introduce a bilevel optimization scheme coupled with a mutual information estimator for irregular graphs. Moreover, we propose a continuous relaxation for subgraph selection with a connectivity loss for stabilization. We further theoretically prove the error bound of our estimation scheme for mutual information and the noise-invariant nature of IB-subgraph. Extensive experiments on graph learning and large-scale point cloud tasks demonstrate the superior property of IB-subgraph.

An experimental study of neural estimators of the mutual information between random vectors modeling power spectrum features

Mutual information (MI) quantifies the statistical dependency between a pair of random variables and plays a central role in signal processing and data analysis. Recent advances in machine learning have enabled the estimation of MI from a dataset using the expressive power of neural networks. In this study, we conducted a comparative experimental analysis of several existing neural estimators of MI between random vectors that model power spectrum features. We explored alternative models of power spectrum features by leveraging information-theoretic data processing inequality and bijective transformations. Empirical results demonstrated that each neural estimator of MI covered in this study has its limitations. In practical applications, we recommend the collective use of existing neural estimators in a complementary manner for the problem of estimating MI between power spectrum features.

Probabilistic Prognosis of Wind Turbine Faults With Feature Selection and Confidence Calibration

With the ever-increasing expansion of the installed capacity of wind power generation, reliable condition monitoring for wind turbines (WTs) has become increasingly important. To this end, this paper proposes a probabilistic WT fault prognosis (WTFP) scheme to output reliable fault probabilities in addition to class predictions. First, a multivariate time series (MTS) based mutual information estimator (MIE) is developed to integrate with combinational optimization, selecting features that contain more beneficial temporal data patterns for WTFP. Then, a multi-fault prognosis model for WTs is trained based on the selected features and MTS learning network. Next, a confidence calibration (CC) post-processing module is appended to re-construct the mapping relation between network logits and posterior probabilities by minimizing negative log-likelihood, thereby calibrating confidence estimates to approximate the true correctness likelihood as much as possible while keeping the high accuracy of the trained MTS learning network. These components finally drive a comprehensive WTFP model. Test results on the field data of WT verify the efficacy of MTS-based MIE and CC, and show that the proposed probabilistic WTFP scheme can provide more reliable probability estimates, contributing to better decision-making.

Hierarchically Contrastive Hard Sample Mining for Graph Self-Supervised Pretraining.

Contrastive learning has recently emerged as a powerful technique for graph self-supervised pretraining (GSP). By maximizing the mutual information (MI) between a positive sample pair, the network is forced to extract discriminative information from graphs to generate high-quality sample representations. However, we observe that, in the process of MI maximization (Infomax), the existing contrastive GSP algorithms suffer from at least one of the following problems: 1) treat all samples equally during optimization and 2) fall into a single contrasting pattern within the graph. Consequently, the vast number of well-categorized samples overwhelms the representation learning process, and limited information is accumulated, thus deteriorating the learning capability of the network. To solve these issues, in this article, by fusing the information from different views and conducting hard sample mining in a hierarchically contrastive manner, we propose a novel GSP algorithm called hierarchically contrastive hard sample mining (HCHSM). The hierarchical property of this algorithm is manifested in two aspects. First, according to the results of multilevel MI estimation in different views, the MI-based hard sample selection (MHSS) module keeps filtering the easy nodes and drives the network to focus more on hard nodes. Second, to collect more comprehensive information for hard sample learning, we introduce a hierarchically contrastive scheme to sequentially force the learned node representations to involve multilevel intrinsic graph features. In this way, as the contrastive granularity goes finer, the complementary information from different levels can be uniformly encoded to boost the discrimination of hard samples and enhance the quality of the learned graph embedding. Extensive experiments on seven benchmark datasets indicate that the HCHSM performs better than other competitors on node classification and node clustering tasks. The source code of HCHSM is available at https://github.com/WxTu/HCHSM.