Mutual Information Estimation Research Articles

Unified ISAC Pareto Boundary Based on Mutual Information and Minimum Mean-Square Error Estimation

Unified ISAC Pareto Boundary Based on Mutual Information and Minimum Mean-Square Error Estimation

Training multi-source domain adaptation network by mutual information estimation and minimization

We address the problem of Multi-Source Domain Adaptation (MSDA), which trains a neural network using multiple labeled source datasets and an unlabeled target dataset, and expects the trained network to well classify the unlabeled target data. The main challenge in this problem is that the datasets are generated by relevant but different joint distributions. In this paper, we propose to address this challenge by estimating and minimizing the mutual information in the network latent feature space, which leads to the alignment of the source joint distributions and target joint distribution simultaneously. Here, the estimation of the mutual information is formulated into a convex optimization problem, such that the global optimal solution can be easily found. We conduct experiments on several public datasets, and show that our algorithm statistically outperforms its competitors. Video and code are available at https://github.com/sentaochen/Mutual-Information-Estimation-and-Minimization.

Stable and Fast Deep Mutual Information Maximization Based on Wasserstein Distance.

Deep learning is one of the most exciting and promising techniques in the field of artificial intelligence (AI), which drives AI applications to be more intelligent and comprehensive. However, existing deep learning techniques usually require a large amount of expensive labeled data, which limit the application and development of deep learning techniques, and thus it is imperative to study unsupervised machine learning. The learning of deep representations by mutual information estimation and maximization (Deep InfoMax or DIM) method has achieved unprecedented results in the field of unsupervised learning. However, in the DIM method, to restrict the encoder to learn more normalized feature representations, an adversarial network learning method is used to make the encoder output consistent with a priori positively distributed data. As we know, the model training of the adversarial network learning method is difficult to converge, because there is a logarithmic function in the loss function of the cross-entropy measure, and the gradient of the model parameters is susceptible to the "gradient explosion" or "gradient disappearance" phenomena, which makes the training of the DIM method extremely unstable. In this regard, we propose a Wasserstein distance-based DIM method to solve the stability problem of model training, and our method is called the WDIM. Subsequently, the training stability of the WDIM method and the classification ability of unsupervised learning are verified on the CIFAR10, CIFAR100, and STL10 datasets. The experiments show that our proposed WDIM method is more stable to parameter updates, has faster model convergence, and at the same time, has almost the same accuracy as the DIM method on the classification task of unsupervised learning. Finally, we also propose a reflection of future research for the WDIM method, aiming to provide a research idea and direction for solving the image classification task with unsupervised learning.

BannMI deciphers potential n-to-1 information transduction in signaling pathways to unravel message of intrinsic apoptosis.

Cell fate decisions, such as apoptosis or proliferation, are communicated via signaling pathways. The pathways are heavily intertwined and often consist of sequential interaction of proteins (kinases). Information integration takes place on the protein level via n-to-1 interactions. A state-of-the-art procedure to quantify information flow (edges) between signaling proteins (nodes) is network inference. However, edge weight calculation typically refers to 1-to-1 interactions only and relies on mean protein phosphorylation levels instead of single cell distributions. Information theoretic measures such as the mutual information (MI) have the potential to overcome these shortcomings but are still rarely used. This work proposes a Bayesian nearest neighbor-based MI estimator (BannMI) to quantify n-to-1 kinase dependency in signaling pathways. BannMI outperforms the state-of-the-art MI estimator on protein-like data in terms of mean squared error and Pearson correlation. Using BannMI, we analyze apoptotic signaling in phosphoproteomic cancerous and noncancerous breast cell line data. Our work provides evidence for cooperative signaling of several kinases in programmed cell death and identifies a potential key role of the mitogen-activated protein kinase p38. Source code and applications are available at: https://github.com/zuiop11/nn_info and can be downloaded via Pip as Python package: nn-info.

FastMI: A fast and consistent copula-based nonparametric estimator of mutual information

As a fundamental concept in information theory, mutual information (MI) has been commonly applied to quantify association between random vectors. Most existing nonparametric estimators of MI have unstable statistical performance since they involve parameter tuning. We develop a consistent and powerful estimator, called fastMI, that does not incur any parameter tuning. Based on a copula formulation, fastMI estimates MI by leveraging Fast Fourier transform-based estimation of the underlying density. Extensive simulation studies reveal that fastMI outperforms state-of-the-art estimators with improved estimation accuracy and reduced run time for large data sets. fastMI provides a powerful test for independence that exhibits satisfactory type I error control. Anticipating that it will be a powerful tool in estimating mutual information in a broad range of data, we develop an R package fastMI for broader dissemination.

Estimating Mutual Information for Spike Trains: A Bird Song Example.

Zebra finches are a model animal used in the study of audition. They are adept at recognizing zebra finch songs, and the neural pathway involved in song recognition is well studied. Here, this example is used to illustrate the estimation of mutual information between stimuli and responses using a Kozachenko-Leonenko estimator. The challenge in calculating mutual information for spike trains is that there are no obvious coordinates for the data. The Kozachenko-Leonenko estimator does not require coordinates; it relies only on the distance between data points. In the case of bird songs, estimating the mutual information demonstrates that the information content of spiking does not diminish as the song progresses.

Capacity-driven end-to-end learning approach for NOMA with finite-alphabet inputs

Optimal Non-orthogonal multiple access (NOMA) design with practical finite-alphabet inputs instead of ideal Gaussian inputs has been addressed in this paper. With the aid of tight mutual information neural estimation, a capacity-driven end-to-end learning communication framework is proposed to acquire optimal superposition coding and thus approach the capacity region of NOMA with finite-alphabet inputs. For such a purpose, we firstly give a closed-form expression of conditional mutual information to define the capacity region of NOMA with finite-alphabet inputs. A natural tight bound on mutual information is then derived for simplifying the closed-form expression, which is needed to establish a consistent and tight mutual information neural estimator. Lastly, in the end-to-end learning framework, an encoder and a self-attention, which are respectively responsible for constellation learning and power allocation, are guided by the capacity approaching method based on mutual information estimation, while successive interference cancellation (SIC) is designed as a neural unit embedded in the decoder to guarantee decoding performance. Numerical results show that the proposed capacity-driven end-to-end learning scheme achieves the achievable rate region for NOMA, with SER performance comparable to state-of-the-art decoding schemes.

Smoothed noise contrastive mutual information neural estimation

Information Noise Contrastive Estimation (InfoNCE) is a popular neural estimator of mutual information (MI). While InfoNCE has demonstrated impressive results in representation learning, the estimation can be significantly off. While the original estimator is known to underestimate the MI due to the logn upper bound, where n is the sample size, we show that some subsequent fix can cause the MI estimate to overshoot apparently without any bound. We propose a novel MI variational estimator, smoothed InfoNCE, that resolves the issues by smoothing out the contrastive estimation. Experiments on high-dimensional Gaussian data confirm that the proposed estimate can break the logn curse without overshooting.

Improved Swarm Intelligence Optimization Model with Mutual Information Estimation for Feature Selection in Microarray Bioinformatics Datasets for Diseases Diagnosis

Improved Swarm Intelligence Optimization Model with Mutual Information Estimation for Feature Selection in Microarray Bioinformatics Datasets for Diseases Diagnosis

Nearest-Neighbor Sampling Based Conditional Independence Testing

The conditional randomization test (CRT) was recently proposed to test whether two random variables X and Y are conditionally independent given random variables Z. The CRT assumes that the conditional distribution of X given Z is known under the null hypothesis and then it is compared to the distribution of the observed samples of the original data. The aim of this paper is to develop a novel alternative of CRT by using nearest-neighbor sampling without assuming the exact form of the distribution of X given Z. Specifically, we utilize the computationally efficient 1-nearest-neighbor to approximate the conditional distribution that encodes the null hypothesis. Then, theoretically, we show that the distribution of the generated samples is very close to the true conditional distribution in terms of total variation distance. Furthermore, we take the classifier-based conditional mutual information estimator as our test statistic. The test statistic as an empirical fundamental information theoretic quantity is able to well capture the conditional-dependence feature. We show that our proposed test is computationally very fast, while controlling type I and II errors quite well. Finally, we demonstrate the efficiency of our proposed test in both synthetic and real data analyses.

Invariant Representations with Stochastically Quantized Neural Networks

Representation learning algorithms offer the opportunity to learn invariant representations of the input data with regard to nuisance factors. Many authors have leveraged such strategies to learn fair representations, i.e., vectors where information about sensitive attributes is removed. These methods are attractive as they may be interpreted as minimizing the mutual information between a neural layer's activations and a sensitive attribute. However, the theoretical grounding of such methods relies either on the computation of infinitely accurate adversaries or on minimizing a variational upper bound of a mutual information estimate. In this paper, we propose a methodology for direct computation of the mutual information between neurons in a layer and a sensitive attribute. We employ stochastically-activated binary neural networks, which lets us treat neurons as random variables. Our method is therefore able to minimize an upper bound on the mutual information between the neural representations and a sensitive attribute. We show that this method compares favorably with the state of the art in fair representation learning and that the learned representations display a higher level of invariance compared to full-precision neural networks.

Near-Optimal Learning of Tree-Structured Distributions by Chow and Liu

We provide finite sample guarantees for the classical Chow–Liu algorithm [Chow and Liu, IEEE Trans. Inform. Theory, 14 (1968), pp. 462–467] to learn a tree-structured graphical model of a distribution. For a distribution on and a tree on nodes, we say is an -approximate tree for if there is a -structured distribution such that is at most more than the best possible tree-structured distribution for . We show that if itself is tree-structured, then the Chow–Liu algorithm with the plug-in estimator for mutual information with independent and identically distributed samples outputs an -approximate tree for with constant probability. In contrast, for a general (which may not be tree-structured), samples are necessary to find an -approximate tree. Our upper bound is based on a new conditional independence tester that addresses an open problem posed by Canonne et al. [Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, ACM, 2018, pp. 735–748]: we prove that for three random variables each over , testing if is 0 or is possible with samples. Finally, we show that for a specific tree , with samples from a distribution over , one can efficiently learn the closest -structured distribution in KL divergence by applying the add-1 estimator at each node.

A Firefly based Deep Belief Signal Specification based Novel Hybrid Technique for EEG Signal Analysis

In the medical sector, electroencephalogram (EEG) signal analysis is a fascinating and indispensable diagnostic tool for brain diseases such as epilepsy. Neurobiological diseases are difficult and time-consuming to diagnose due to the excessive associated noise in the EEG signal, which can significantly diminish and impede accurate predictions of such diseases. Existing techniques rely on complex processing blocks, such as Granger causality, chaos analysis, mutual information estimation, etc., in order to function satisfactorily. This paper proposes the development of a novel EEG signal analysis system employing Deep Belief Signal Specification (FbDBSS) for the diagnosis of Schizophrenia (SZ). This signal is used to initially train the system alongside conventional EEG, followed by preprocessing. The resultant signal is sent to the classifier, which extracts its characteristics for use in the final analysis. MATLAB is used to implement the proposed design for determining the robustness score in terms of accuracy, precision, F-measure, recall (sensitivity), execution time, and power consumption. Achieving an accuracy of 97%, a precision of 96.8%, an F-measurement of 96%, a recall rate (i.e. sensitivity) of 97%, an execution time of 19 s, and a power consumption of 5dBW demonstrates the effectiveness of the proposed method for EEG signal analysis in health care monitoring.

Analysis of Deep Convolutional Neural Networks Using Tensor Kernels and Matrix-Based Entropy.

Analyzing deep neural networks (DNNs) via information plane (IP) theory has gained tremendous attention recently to gain insight into, among others, DNNs' generalization ability. However, it is by no means obvious how to estimate the mutual information (MI) between each hidden layer and the input/desired output to construct the IP. For instance, hidden layers with many neurons require MI estimators with robustness toward the high dimensionality associated with such layers. MI estimators should also be able to handle convolutional layers while at the same time being computationally tractable to scale to large networks. Existing IP methods have not been able to study truly deep convolutional neural networks (CNNs). We propose an IP analysis using the new matrix-based Rényi's entropy coupled with tensor kernels, leveraging the power of kernel methods to represent properties of the probability distribution independently of the dimensionality of the data. Our results shed new light on previous studies concerning small-scale DNNs using a completely new approach. We provide a comprehensive IP analysis of large-scale CNNs, investigating the different training phases and providing new insights into the training dynamics of large-scale neural networks.

Benchmarking Neural Capacity Estimation: Viability and Reliability

Recently, several methods have been proposed for estimating the mutual information from sample data using deep neural networks. This approach is referred to as neural mutual information estimation (NMIE). NMIEs differ from other approaches in the literature as they are data-driven estimators. As such, they have the potential to perform well on a large class of capacity problems. To test the performance across various NMIEs, it is desirable to establish a benchmark encompassing the different challenges of capacity estimation. This is the objective of this paper. We consider three scenarios for benchmarking: (i) the classic AWGN channel, (ii) channels continuous inputs – the optical intensity and peak-power constrained AWGN channel (iii) channels with a discrete output – i.e., the Poisson channel. We also consider the extension to the multi-terminal case with (iv) the AWGN and optical MAC models. We argue that benchmarking a certain NMIE across these four scenarios provides a substantive test of performance. We study the performance of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">mutual information neural estimator</i> (MINE), <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">smoothed mutual information lower-bound estimator</i> (SMILE), and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">directed information neural estimator</i> (DINE) and provide insights into the performance of other methods as well. To summarize our benchmarking results, MINE provides the most reliable performance.

A robust estimator of mutual information for deep learning interpretability

We develop the use of mutual information (MI), a well-established metric in information theory, to interpret the inner workings of deep learning (DL) models. To accurately estimate MI from a finite number of samples, we present GMM-MI (pronounced ‘Jimmie’), an algorithm based on Gaussian mixture models that can be applied to both discrete and continuous settings. GMM-MI is computationally efficient, robust to the choice of hyperparameters and provides the uncertainty on the MI estimate due to the finite sample size. We extensively validate GMM-MI on toy data for which the ground truth MI is known, comparing its performance against established MI estimators. We then demonstrate the use of our MI estimator in the context of representation learning, working with synthetic data and physical datasets describing highly non-linear processes. We train DL models to encode high-dimensional data within a meaningful compressed (latent) representation, and use GMM-MI to quantify both the level of disentanglement between the latent variables, and their association with relevant physical quantities, thus unlocking the interpretability of the latent representation. We make GMM-MI publicly available in this GitHub repository.

DeepSE-WF: Unified Security Estimation for Website Fingerprinting Defenses

Website fingerprinting (WF) attacks, usually conducted with the help of a machine learning-based classifier, enable a network eavesdropper to pinpoint which website a user is accessing through the inspection of traffic patterns. These attacks have been shown to succeed even when users browse the Internet through encrypted tunnels, e.g., through Tor or VPNs. To assess the security of new defenses against WF attacks, recent works have proposed feature-dependent theoretical frameworks that estimate the Bayes error of an adversary's features set or the mutual information leaked by manually-crafted features. Unfortunately, as WF attacks increasingly rely on deep learning and latent feature spaces, our experiments show that security estimations based on simpler (and less informative) manually-crafted features can no longer be trusted to assess the potential success of a WF adversary in defeating such defenses. In this work, we propose DeepSE-WF, a novel WF security estimation framework that leverages specialized kNN-based estimators to produce Bayes error and mutual information estimates from learned latent feature spaces, thus bridging the gap between current WF attacks and security estimation methods. Our evaluation reveals that DeepSE-WF produces tighter security estimates than previous frameworks, reducing the required computational resources to output security estimations by one order of magnitude.

Gene regulation network inference using k-nearest neighbor-based mutual information estimation: revisiting an old DREAM

BackgroundA cell exhibits a variety of responses to internal and external cues. These responses are possible, in part, due to the presence of an elaborate gene regulatory network (GRN) in every single cell. In the past 20 years, many groups worked on reconstructing the topological structure of GRNs from large-scale gene expression data using a variety of inference algorithms. Insights gained about participating players in GRNs may ultimately lead to therapeutic benefits. Mutual information (MI) is a widely used metric within this inference/reconstruction pipeline as it can detect any correlation (linear and non-linear) between any number of variables (n-dimensions). However, the use of MI with continuous data (for example, normalized fluorescence intensity measurement of gene expression levels) is sensitive to data size, correlation strength and underlying distributions, and often requires laborious and, at times, ad hoc optimization.ResultsIn this work, we first show that estimating MI of a bi- and tri-variate Gaussian distribution using k-nearest neighbor (kNN) MI estimation results in significant error reduction as compared to commonly used methods based on fixed binning. Second, we demonstrate that implementing the MI-based kNN Kraskov–Stoögbauer–Grassberger (KSG) algorithm leads to a significant improvement in GRN reconstruction for popular inference algorithms, such as Context Likelihood of Relatedness (CLR). Finally, through extensive in-silico benchmarking we show that a new inference algorithm CMIA (Conditional Mutual Information Augmentation), inspired by CLR, in combination with the KSG-MI estimator, outperforms commonly used methods.ConclusionsUsing three canonical datasets containing 15 synthetic networks, the newly developed method for GRN reconstruction—which combines CMIA, and the KSG-MI estimator—achieves an improvement of 20–35% in precision-recall measures over the current gold standard in the field. This new method will enable researchers to discover new gene interactions or better choose gene candidates for experimental validations.

Lowering levels of reelin in entorhinal cortex layer II-neurons results in lowered levels of intracellular amyloid-β.

Projection neurons in the anteriolateral part of entorhinal cortex layer II are the predominant cortical site for hyper-phosphorylation of tau and formation of neurofibrillary tangles in prodromal Alzheimer's disease. A majority of layer II projection neurons in anteriolateral entorhinal cortex are unique among cortical excitatory neurons by expressing the protein reelin. In prodromal Alzheimer's disease, these reelin-expressing neurons are prone to accumulate intracellular amyloid-β, which is mimicked in a rat model that replicates the spatio-temporal cascade of the disease. Two important findings in relation to this are that reelin-signalling downregulates tau phosphorylation, and that oligomeric amyloid-β interferes with reelin-signalling. Taking advantage of this rat model, we used proximity ligation assay to assess whether reelin and intracellular amyloid-β directly interact during early, pre-plaque stages in anteriolateral entorhinal cortex layer II reelin-expressing neurons. We next made a viral vector delivering micro-RNA against reelin, along with a control vector, and infected reelin-expressing anteriolateral entorhinal cortex layer II-neurons to test whether reelin levels affect levels of intracellular amyloid-β and/or amyloid precursor protein. We analysed 25.548 neurons from 24 animals, which results in three important findings. First, in reelin-expressing anteriolateral entorhinal cortex layer II-neurons, reelin and intracellular amyloid-β engage in a direct protein-protein interaction. Second, injecting micro-RNA against reelin lowers reelin levels in these neurons, amounting to an effect size of 1.3-4.5 (Bayesian estimation of Cohen's d effect size, 95% credible interval). This causes a concomitant reduction of intracellular amyloid-β ranging across three levels of aggregation, including a reduction of Aβ42 monomers/dimers amounting to an effect size of 0.5-3.1, a reduction of Aβ prefibrils amounting to an effect size of 1.1-3.5 and a reduction of protofibrils amounting to an effect size of 0.05-2.1. Analysing these data using Bayesian estimation of mutual information furthermore reveals that levels of amyloid-β are dependent on levels of reelin. Third, the reduction of intracellular amyloid-β occurs without any substantial associated changes in levels of amyloid precursor protein. We conclude that reelin and amyloid-β directly interact at the intracellular level in the uniquely reelin-expressing projection neurons in anteriolateral entorhinal cortex layer II, where levels of amyloid-β are dependent on levels of reelin. Since amyloid-β is known to impair reelin-signalling causing upregulated phosphorylation of tau, our findings are likely relevant to the vulnerability for neurofibrillary tangle-formation of this entorhinal neuronal population.

Short Blocklength Wiretap Channel Codes via Deep Learning: Design and Performance Evaluation

We design short blocklength codes for the Gaussian wiretap channel under information-theoretic security guarantees. Our approach consists in decoupling the reliability and secrecy constraints in our code design. Specifically, we handle the reliability constraint via an autoencoder, and handle the secrecy constraint with hash functions. For blocklengths smaller than or equal to 128, we evaluate through simulations the probability of error at the legitimate receiver and the leakage at the eavesdropper for our code construction. This leakage is defined as the mutual information between the confidential message and the eavesdropper’s channel observations, and is empirically measured via a neural network-based mutual information estimator. Our simulation results provide examples of codes with positive secrecy rates that outperform the best known achievable secrecy rates obtained non-constructively for the Gaussian wiretap channel. Additionally, we show that our code design is suitable for the compound and arbitrarily varying Gaussian wiretap channels, for which the channel statistics are not perfectly known but only known to belong to a pre-specified uncertainty set. These models not only capture uncertainty related to channel statistics estimation, but also scenarios where the eavesdropper jams the legitimate transmission or influences its own channel statistics by changing its location.