Mutual Information Estimation Research Articles

Neural Estimators for Conditional Mutual Information Using Nearest Neighbors Sampling

The estimation of mutual information (MI) or conditional mutual information (CMI) from a set of samples is a long-standing problem. A recent line of work in this area has leveraged the approximation power of artificial neural networks and has shown improvements over conventional methods. One important challenge in this new approach is the need to obtain, given the original dataset, a different set where the samples are distributed according to a specific product density function. This is particularly challenging when estimating CMI. In this paper, we introduce a new technique, based on k nearest neighbors (k-NN), to perform the resampling and derive high-confidence concentration bounds for the sample average. Then the technique is employed to train a neural network classifier and the CMI is estimated accordingly. We propose three estimators using this technique and prove their consistency, make a comparison between them and similar approaches in the literature, and experimentally show improvements in estimating the CMI in terms of accuracy and variance of the estimators.

On the Use of Correlation and MI as a Measure of Metabolite-Metabolite Association for Network Differential Connectivity Analysis.

Metabolite differential connectivity analysis has been successful in investigating potential molecular mechanisms underlying different conditions in biological systems. Correlation and Mutual Information (MI) are two of the most common measures to quantify association and for building metabolite—metabolite association networks and to calculate differential connectivity. In this study, we investigated the performance of correlation and MI to identify significantly differentially connected metabolites. These association measures were compared on (i) 23 publicly available metabolomic data sets and 7 data sets from other fields, (ii) simulated data with known correlation structures, and (iii) data generated using a dynamic metabolic model to simulate real-life observed metabolite concentration profiles. In all cases, we found more differentially connected metabolites when using correlation indices as a measure for association than MI. We also observed that different MI estimation algorithms resulted in difference in performance when applied to data generated using a dynamic model. We concluded that there is no significant benefit in using MI as a replacement for standard Pearson’s or Spearman’s correlation when the application is to quantify and detect differentially connected metabolites.

Averaging Is Probably Not the Optimum Way of Aggregating Parameters in Federated Learning

Federated learning is a decentralized topology of deep learning, that trains a shared model through data distributed among each client (like mobile phones, wearable devices), in order to ensure data privacy by avoiding raw data exposed in data center (server). After each client computes a new model parameter by stochastic gradient descent (SGD) based on their own local data, these locally-computed parameters will be aggregated to generate an updated global model. Many current state-of-the-art studies aggregate different client-computed parameters by averaging them, but none theoretically explains why averaging parameters is a good approach. In this paper, we treat each client computed parameter as a random vector because of the stochastic properties of SGD, and estimate mutual information between two client computed parameters at different training phases using two methods in two learning tasks. The results confirm the correlation between different clients and show an increasing trend of mutual information with training iteration. However, when we further compute the distance between client computed parameters, we find that parameters are getting more correlated while not getting closer. This phenomenon suggests that averaging parameters may not be the optimum way of aggregating trained parameters.

Zero-shot learning by mutual information estimation and maximization

The key of zero-shot learning is to use the visual-semantic embedding to transfer the knowledge from seen classes to unseen classes. In this paper, we propose to build the visual-semantic embedding by maximizing the mutual information between visual features and corresponding attributes. Then, the mutual information between visual and semantic features can be utilized to guide the knowledge transfer from seen domain to unseen domain. Since we are primarily interested in maximizing mutual information, we introduce the noise-contrastive estimation to calculate lower-bound value of mutual information. Through the noise-contrastive estimation, we reformulate zero-shot learning as a binary classification problem, i.e., classifying the matching visual-semantic pairs (positive samples) and mismatching visual-semantic pairs (negative/noise samples). Experiments conducted on five datasets demonstrate that the proposed mutual information estimators outperforms current state-of-the-art methods both in conventional and generalized zero-shot learning settings.

Relationship Between Kendall's tau Correlation and Mutual Information

Mutual information (MI) can be viewed as a measure of multivariate association in a random vector. However, the estimation of MI is difficult since the estimation of the joint probability density function (PDF) of non Gaussian distributed data is a hard problem. Copula function is an appropriate tool for estimating MI since the joint probability density function ofrandom variables can be expressed as the product of the associated copula density function and marginal PDF’s. With a little search, we find that the proposed copulas-based mutual information is much more accurate than conventional methods such as the joint histogram and Parzen window-based MI. In this paper, by using the copulas-based method, we compute MI forsome family of bivariate distribution functions and study the relationship between Kendall’s tau correlation and MI of bivariate distributions. Finally, using a real dataset, we illustrate the efficiency of this approach.

Weighted-averaging-based classification of laser-induced breakdown spectroscopy measurements using most informative spectral lines

In this study, efficient spectral line selection and weighted-averaging-based processing schemes are proposed for the classification of laser-induced breakdown spectroscopy (LIBS) measurements. For fast on-line classification, a set of representative spectral lines are selected and processed relying on the information metric, instead of the time consuming full spectrum based analysis. The most informative spectral line sets are investigated by the joint mutual information estimation (MIE) evaluated with the Gaussian kernel density, where dominant intensity peaks associated with the concentrated components are not necessarily most valuable for classification. In order to further distinguish the characteristic patterns of the LIBS measured spectrum, two-dimensional spectral images are synthesized through column-wise concatenation of the peaks along with their neighbors. For fast classification while preserving the effect of distinctive peak patterns, column-wise Gaussian weighted averaging is applied to the synthesized images, yielding a favorable trade-off between classification performance and computational complexity. To explore the applicability of the proposed schemes, two applications of alloy classification and skin cancer detection are investigated with the multi-class and binary support vector machines classifiers, respectively. The MIE measures associated with selected spectral lines in both applications show a strong correlation to the actual classification or detection accuracy, which enables to find out meaningful combinations of spectral lines. In addition, the peak patterns of the selected lines and their Gaussian weighted averaging with neighbors of the selected peaks efficiently distinguish different classes of LIBS measured spectrum.

Analysis of KNN Information Estimators for Smooth Distributions

KSG mutual information estimator, which is based on the distances of each sample to its k-th nearest neighbor, is widely used to estimate mutual information between two continuous random variables. Existing work has analyzed the convergence rate of this estimator for random variables whose densities are bounded away from zero in its support. In practice, however, KSG estimator also performs well for a much broader class of distributions, including not only those with bounded support and densities bounded away from zero, but also those with bounded support but densities approaching zero, and those with unbounded support. In this paper, we analyze the convergence rate of the error of KSG estimator for smooth distributions, whose support of density can be both bounded and unbounded. As KSG mutual information estimator can be viewed as an adaptive recombination of KL entropy estimators, in our analysis, we also provide convergence analysis of KL entropy estimator for a broad class of distributions.

Variational approach for privacy funnel optimization on continuous data

Here we consider a common data encryption problem encountered by users who want to disclose some data to gain utility but preserve their private information. Specifically, we consider the inference attack, in which an adversary conducts inference on the disclosed data to gain information about users’ private data. Following privacy funnel (Makhdoumi et al., 2014), assuming that the original data X is transformed into Z before disclosing and the log loss is used for both privacy and utility metrics, then the problem can be modeled as finding a mapping X→Z that maximizes mutual information between X and Z subject to a constraint that the mutual information between Z and private data S is smaller than a predefined threshold ϵ. In contrast to the original study (Makhdoumi et al., 2014), which only focused on discrete data, we consider the more general and practical setting of continuous and high-dimensional disclosed data (e.g., image data). Most previous work on privacy-preserving representation learning is based on adversarial learning or generative adversarial networks, which has been shown to suffer from the vanishing gradient problem, and it is experimentally difficult to eliminate the relationship with private data Y when Z is constrained to retain more information about X. Here we propose a simple but effective variational approach that does not rely on adversarial training. Our experimental results show that our approach is stable and outperforms previous methods in terms of both downstream task accuracy and mutual information estimation.

Estimation of mutual information for real-valued data with error bars and controlled bias.

Estimation of mutual information between (multidimensional) real-valued variables is used in analysis of complex systems, biological systems, and recently also quantum systems. This estimation is a hard problem, and universally good estimators provably do not exist. We focus on the estimator introduced by Kraskov etal. [Phys. Rev. E 69, 066138 (2004)PLEEE81539-375510.1103/PhysRevE.69.066138] based on the statistics of distances between neighboring data points, which empirically works for a wide class of underlying probability distributions. First, we illustrate pitfalls of naively applying bootstrapping to estimate the variance of the mutual information estimate. Then we improve this estimator by (1) expanding its range of applicability and by providing (2) a self-consistent way of verifying the absence of bias, (3) a method for estimation of its variance, and (4) guidelines for choosing the values of the free parameter of the estimator. We demonstrate the performance of our estimator on synthetic data sets, as well as on neurophysiological and systems biology data sets.

Empirical Estimation of Information Measures: A Literature Guide

We give a brief survey of the literature on the empirical estimation of entropy, differential entropy, relative entropy, mutual information and related information measures. While those quantities are of central importance in information theory, universal algorithms for their estimation are increasingly important in data science, machine learning, biology, neuroscience, economics, language, and other experimental sciences.

On the optimal number estimation of selected features using joint histogram based mutual information for speech emotion recognition

Mutual information (MI) has been previously used to select the relevant features for the task of speech emotion recognition (SER). However, the procedure does not deliver the optimal number of relevant features. We propose MI based criterion for estimating this number defined as the minimum number of features that explains the variable of the class indices. In order to minimize the MI estimation errors, we also search the best histogram binning choice considering three formulas: Sturges, Scott and LMSE. Four selection strategies MMI, CMI, JMI and TMI have been implemented and applied on 39-features vectors and on large dimension vectors. The feature selection results have been validated on independent text SER system, based on GMM classifier and evaluated on EMO-db database. Results demonstrate that LMSE bin choice gives the best MI estimation and ensures a minimal number of features with slight performance drop. Particularly, using the proposed stopping criterion, the CMI strategy achieves reduction of 48.72% in the case of the 39-features vectors size and 67.86% in the case of large dimension vectors. Moreover, using the recognition rate criterion, the JMI strategy gives a comparable feature reduction with slight improvement of performance but requiring very high computation capabilities.

Non-destructive diagnostic of aircraft engine blades by Fuzzy Decision Tree

A new algorithm for non-destructive diagnosis based on signal classification is proposed. This algorithm permits classifying objects with different properties depending on signals characterizing the objects. In this paper, the algorithm is applied to diagnose blades of aircraft engine gas turbine according to vibration signal after a non-destructive shock excitation and to classify them as defective and faultless. Just like other signal classification algorithms, this algorithm consists of two steps that are signal preliminary transformation and classification. However, these steps are modified in the proposed algorithm. Unlike other algorithms, the fuzzy classifier is used in the classification step. The change of the classifier type causes modification of the preliminary transformation step because the attributes for the classification must be fuzzy. Therefore, fuzzification procedure is added into the step of preliminary transformation. The fuzzy classifier for the problem of aircraft engine blades diagnosis is ordered Fuzzy Decision Tree (oFDT) that is inducted by estimation of Cumulative Mutual Information. This induction has good efficiency for a small set of initial data. The accuracy of the classification of defective and faultless blades for the proposed algorithm is 98.5%, and oFDT for this classification is inducted based on 32 signals only. The comparison with other classification algorithms shows that oFDT based algorithm considered in this paper gives the best result for this problem.

Information-theoretic analysis of multivariate single-cell signaling responses.

Mathematical methods of information theory appear to provide a useful language to describe how stimuli are encoded in activities of signaling effectors. Exploring the information-theoretic perspective, however, remains conceptually, experimentally and computationally challenging. Specifically, existing computational tools enable efficient analysis of relatively simple systems, usually with one input and output only. Moreover, their robust and readily applicable implementations are missing. Here, we propose a novel algorithm, SLEMI—statistical learning based estimation of mutual information, to analyze signaling systems with high-dimensional outputs and a large number of input values. Our approach is efficient in terms of computational time as well as sample size needed for accurate estimation. Analysis of the NF-κB single—cell signaling responses to TNF-α reveals that NF-κB signaling dynamics improves discrimination of high concentrations of TNF-α with a relatively modest impact on discrimination of low concentrations. Provided R-package allows the approach to be used by computational biologists with only elementary knowledge of information theory.

Nonparametric independence testing via mutual information

SummaryWe propose a test of independence of two multivariate random vectors, given a sample from the underlying population. Our approach is based on the estimation of mutual information, whose decomposition into joint and marginal entropies facilitates the use of recently developed efficient entropy estimators derived from nearest neighbour distances. The proposed critical values may be obtained by simulation in the case where an approximation to one marginal is available or by permuting the data otherwise. This facilitates size guarantees, and we provide local power analyses, uniformly over classes of densities whose mutual information satisfies a lower bound. Our ideas may be extended to provide new goodness-of-fit tests for normal linear models based on assessing the independence of our vector of covariates and an appropriately defined notion of an error vector. The theory is supported by numerical studies on both simulated and real data.

Fingerprint Entropy and Identification Capacity Estimation Based on Pixel-Level Generative Modelling

A family of texture-based generative models for fingerprint images is proposed. The generative models are used to estimate upper bounds on the image entropy for systems with small sensor acquisition. The identification capacity of such systems is then estimated using the mutual information between different samples from the same finger. Similar to the generative model for entropy estimation, pixel-level model families are proposed for estimating the similarity between fingerprint images with a given global affine transformation. These models are used for mutual information estimation, and are also adopted to compensate for local deformations between samples. Finally, it is shown that sensor sizes as small as $52\times 52$ pixels are potentially sufficient to discriminate populations as large as the entire world population that ever lived, given that the complexity-unconstrained recognition algorithm is available which operates on the lowest possible pixel level.

Improved Neural Networks Based on Mutual Information via Information Geometry

This paper presents a new algorithm based on the theory of mutual information and information geometry. This algorithm places emphasis on adaptive mutual information estimation and maximum likelihood estimation. With the theory of information geometry, we adjust the mutual information along the geodesic line. Finally, we evaluate our proposal using empirical datasets that are dedicated for classification and regression. The results show that our algorithm contributes to a significant improvement over existing methods.

Non-parametric estimation of copula based mutual information

Mutual information is a measure for investigating the dependence between two random variables. The copula based estimation of mutual information reduces the complexity because it is depend only on the copula density. We propose two estimators and discuss the asymptotic properties. To compare the performance of the estimators a simulation study is carried out. The methods are illustrated using real data sets.

Copula index for detecting dependence and monotonicity between stochastic signals

This paper introduces a nonparametric copula-based index for detecting the strength and monotonicity structure of linear and nonlinear statistical dependence between pairs of random variables or stochastic signals. Our index, termed Copula Index for Detecting Dependence and Monotonicity (CIM), satisfies several desirable properties of measures of association, including most of Rényi’s properties, the data processing inequality (DPI), and consequently self-equitability. Synthetic data simulations reveal that the statistical power of CIM compares favorably to other state-of-the-art measures of association that are proven to satisfy the DPI. Simulation results with real-world data reveal CIM’s unique ability to detect the monotonicity structure among stochastic signals to find interesting dependencies in large datasets. Additionally, simulations show that CIM compares favorably to estimators of mutual information when discovering Markov network structure.

Unsupervised sentence representations as word information series: Revisiting TF–IDF

Sentence representation at the semantic level is a challenging task for natural language processing and Artificial Intelligence. Despite the advances in word embeddings (i.e. word vector representations), capturing sentence meaning is an open question due to complexities of semantic interactions among words. In this paper, we present an embedding method, which is aimed at learning unsupervised sentence representations from unlabeled text. We propose an unsupervised method that models a sentence as a weighted series of word embeddings. The weights of the series are fitted by using Shannon’s Mutual Information (MI) among words, sentences and the corpus. In fact, the Term Frequency–Inverse Document Frequency transform (TF–IDF) is a reliable estimate of such MI. Our method offers advantages over existing ones: identifiable modules, short-term training, online inference of (unseen) sentence representations, as well as independence from domain, external knowledge and linguistic annotation resources. Results showed that our model, despite its concreteness and low computational cost, was competitive with the state of the art in well-known Semantic Textual Similarity (STS) tasks.

Dissecting alterations in human CD8+ T cells with aging by high-dimensional single cell mass cytometry

We investigated the effect of aging on the multi-dimensional characteristics and heterogeneity of human peripheral CD8+ T cells defined by the expression of a set of molecules at the single cell level using the recently developed mass cytometry or Cytometry by Time-Of-Flight (CyTOF) and computational algorithms. CD8+ T cells of young and older adults had differential expression of molecules, especially those related to cell activation and migration, permitting the clustering of young and older adults through an unbiased approach. The changes in the expression of individual molecules were collectively reflected in the altered high-dimensional profiles of CD8+ T cells in older adults as visualized by the dimensionality reduction analysis tools principal component analysis (PCA) and t-distributed stochastic neighbor embedding (t-SNE). A combination of PhenoGraph clustering and t-SNE analysis revealed heterogeneous subsets of CD8+ T cells that altered with aging. Furthermore, intermolecular quantitative relationships in CD8+ T cells appeared to change with age as determined by the computational algorithm conditional-Density Resampled Estimate of Mutual Information (DREMI). The results of our study showed that heterogeneity, multidimensional characteristics, and intermolecular quantitative relationships in human CD8+ T cells altered with age, distinctively clustering young and older adults through an unbiased approach.