Mutual Information Estimation Research Articles

Equitability, mutual information, and the maximal information coefficient.

How should one quantify the strength of association between two random variables without bias for relationships of a specific form? Despite its conceptual simplicity, this notion of statistical "equitability" has yet to receive a definitive mathematical formalization. Here we argue that equitability is properly formalized by a self-consistency condition closely related to Data Processing Inequality. Mutual information, a fundamental quantity in information theory, is shown to satisfy this equitability criterion. These findings are at odds with the recent work of Reshef et al. [Reshef DN, et al. (2011) Science 334(6062):1518-1524], which proposed an alternative definition of equitability and introduced a new statistic, the "maximal information coefficient" (MIC), said to satisfy equitability in contradistinction to mutual information. These conclusions, however, were supported only with limited simulation evidence, not with mathematical arguments. Upon revisiting these claims, we prove that the mathematical definition of equitability proposed by Reshef et al. cannot be satisfied by any (nontrivial) dependence measure. We also identify artifacts in the reported simulation evidence. When these artifacts are removed, estimates of mutual information are found to be more equitable than estimates of MIC. Mutual information is also observed to have consistently higher statistical power than MIC. We conclude that estimating mutual information provides a natural (and often practical) way to equitably quantify statistical associations in large datasets.

Capacity of Data Collection in Wireless Sensor Networks Based on Mutual Information and MMSE Estimation

We investigate the properties of data collection in wireless sensor networks, in terms of both capacity and power allocation strategy. We consider a scenario in which a number of sensors observe a target being estimated at fusion center (FC) using minimum mean-square error (MMSE) estimator. Based on the relationship between mutual information and MMSE (I-MMSE), the capacity of data collection in coherent and orthogonal multiple access channel (MAC) models is derived. Considering power constraint, the capacity is derived under two scenarios: equal power allocation and optimal power allocation of both models. We provide the upper bound of capacity as a benchmark. In particular, we show that the capacity of data collection scales as Θ((1/2)log(1+L)) when the number of sensors L grows to infinity. We show through simulation results that for both coherent and orthogonal MAC models, the capacity of the optimal power is larger than that of the equal power. We also show that the capacity of coherent MAC is larger than that of orthogonal MAC, particularly when the number of sensors L is large and the total power P is fixed.

Estimation of mutual information by the fuzzy histogram

Mutual Information (MI) is an important dependency measure between random variables, due to its tight connection with information theory. It has numerous applications, both in theory and practice. However, when employed in practice, it is often necessary to estimate the MI from available data. There are several methods to approximate the MI, but arguably one of the simplest and most widespread techniques is the histogram-based approach. This paper suggests the use of fuzzy partitioning for the histogram-based MI estimation. It uses a general form of fuzzy membership functions, which includes the class of crisp membership functions as a special case. It is accordingly shown that the average absolute error of the fuzzy-histogram method is less than that of the naive histogram method. Moreover, the accuracy of our technique is comparable, and in some cases superior to the accuracy of the Kernel density estimation (KDE) method, which is one of the best MI estimation methods. Furthermore, the computational cost of our technique is significantly less than that of the KDE. The new estimation method is investigated from different aspects, such as average error, bias and variance. Moreover, we explore the usefulness of the fuzzy-histogram MI estimator in a real-world bioinformatics application. Our experiments show that, in contrast to the naive histogram MI estimator, the fuzzy-histogram MI estimator is able to reveal all dependencies between the gene-expression data.

Estimation of Mutual Information and Conditional Entropy for Surveillance Optimization

Summary Given a suite of potential surveillance operations, we define surveillance optimization as the problem of choosing the operation that gives the minimum expected value of P90 minus P10 (i.e., P90 – P10) of a specified reservoir variable J (e.g., cumulative oil production) that will be obtained by conditioning J to the observed data. Two questions can be posed: (1) Which surveillance operation is expected to provide the greatest uncertainty reduction in J? and (2) What is the expected value of the reduction in uncertainty that would be achieved if we were to undertake each surveillance operation to collect the associated data and then history match the data obtained? In this work, we extend and apply a conceptual idea that we recently proposed for surveillance optimization to 2D and 3D waterflooding problems. Our method is based on information theory in which the mutual information between J and the random observed data vector Dobs is estimated by use of an ensemble of prior reservoir models. This mutual information reflects the strength of the relationship between J and the potential observed data and provides a qualitative answer to Question 1. Question 2 is answered by calculating the conditional entropy of J to generate an approximation of the expected value of the reduction in (P90 – P10) of J. The reliability of our method depends on obtaining a good estimate of the mutual information. We consider several ways to estimate the mutual information and suggest how a good estimate can be chosen. We validate the results of our proposed method with an exhaustive history-matching procedure. The methodology provides an approximate way to decide the data that should be collected to maximize the uncertainty reduction in a specified reservoir variable and to estimate the reduction in uncertainty that could be obtained. We expect this paper will stimulate significant research on the application of information theory and lead to practical methods and workflows for surveillance optimization.

Computationally Efficient Estimation of Squared-Loss Mutual Information with Multiplicative Kernel Models

Squared-loss mutual information (SMI) is a robust measure of the statistical dependence between random variables. The sample-based SMI approximator called least-squares mutual information (LSMI) was demonstrated to be useful in performing various machine learning tasks such as dimension reduction, clustering, and causal inference. The original LSMI approximates the pointwise mutual information by using the kernel model, which is a linear combination of kernel basis functions located on paired data samples. Although LSMI was proved to achieve the optimal approximation accuracy asymptotically, its approximation capability is limited when the sample size is small due to an insufficient number of kernel basis functions. Increasing the number of kernel basis functions can mitigate this weak

Changes in Seasonal Predictability due to Global Warming

Abstract The change in predictability of monthly mean temperature in a future climate is quantified based on the Community Climate System Model, version 4. According to this model, the North Atlantic overtakes the El Niño–Southern Oscillation (ENSO) as the dominant area of seasonal predictability by 2095. This change arises partly because ENSO becomes less variable and partly because the ENSO teleconnection pattern expands into the Atlantic. Over land, the largest change in temperature predictability occurs in the tropics and is predominantly due to a decrease in ENSO variability. The southern peninsula of Africa and northeast South America are predicted to experience significant drying in a future climate, which decreases the effective heat capacity and memory, and hence increases variance independently of ENSO changes. Extratropical land areas experience enhanced precipitation in a future climate, which decreases temperature variance by the same mechanism. Finally, the model predicts that surface temperatures near the poles will become more predictable and less variable in a future climate, primarily because melting sea ice exposes the underlying sea surface temperature, which is more predictable owing to its longer time scale. Some of these results, especially the change in ENSO variance, are known to be model dependent. This paper also advances the use of information theory to quantify predictability, including 1) deriving a quantitative relation between predictability of the first and second kinds; 2) showing how differences in predictability can be decomposed in two dramatically different ways, facilitating physical interpretation; and 3) proposing a sample estimate of mutual information whose significance can be tested using standard techniques.

Least-squares independence regression for non-linear causal inference under non-Gaussian noise

The discovery of non-linear causal relationship under additive non-Gaussian noise models has attracted considerable attention recently because of their high flexibility. In this paper, we propose a novel causal inference algorithm called least-squares independence regression (LSIR). LSIR learns the additive noise model through the minimization of an estimator of the squared-loss mutual information between inputs and residuals. A notable advantage of LSIR over existing approaches is that tuning parameters such as the kernel width and the regularization parameter can be naturally optimized by cross-validation, allowing us to avoid overfitting in a data-dependent fashion. Through experiments with real-world datasets, we show that LSIR compares favorably with a state-of-the-art causal inference method.

Non–Parametric Estimation of Mutual Information through the Entropy of the Linkage

A new, non–parametric and binless estimator for the mutual information of a d–dimensional random vector is proposed. First of all, an equation that links the mutual information to the entropy of a suitable random vector with uniformly distributed components is deduced. When d = 2 this equation reduces to the well known connection between mutual information and entropy of the copula function associated to the original random variables. Hence, the problem of estimating the mutual information of the original random vector is reduced to the estimation of the entropy of a random vector obtained through a multidimensional transformation. The estimator we propose is a two–step method: first estimate the transformation and obtain the transformed sample, then estimate its entropy. The properties of the new estimator are discussed through simulation examples and its performances are compared to those of the best estimators in the literature. The precision of the estimator converges to values of the same order of magnitude of the best estimator tested. However, the new estimator is unbiased even for larger dimensions and smaller sample sizes, while the other tested estimators show a bias in these cases.

Time-efficient estimation of conditional mutual information for variable selection in classification

An algorithm is proposed for calculating correlation measures based on entropy. The proposed algorithm allows exhaustive exploration of variable subsets on real data. Its time efficiency is demonstrated by comparison against three other variable selection methods based on entropy using 8 data sets from various domains as well as simulated data. The method is applicable to discrete data with a limited number of values making it suitable for medical diagnostic support, DNA sequence analysis, psychometry and other domains.

Dynamic encoding of sensory information in biomimetic sonar baffle

The biosonar system of horseshoe bats stands through several dynamic features that could be related to an outstanding sensory performance. The outer ears (pinnae) of the animals, for example, can change their shapes in a non-rigid fashion and thereby produce qualitative beampattern alterations. Such changes can be reproduced qualitatively with a highly simplified biomimetic baffle prototype. Such a biomimetic prototype is a crucial platform for measuring large amounts of data on the time-variant device behavior that would be difficult to obtain from an animal in vivo. Here, such time-variant data were used to investigate the question whether a dynamic deforming baffle can enhance the encoding of sensory information. This was done based on estimates of the mutual information between beampatterns belonging to different deformation stages of the biomimetic baffle. To make this estimation problem tractable, the transfer functions associated with each direction of sound incidence were sorted into “alphabets” of a small number (X-Y) of discrete signal classes using spectral clustering algorithms. Mutual information estimates computed based on these signal classes indicated very low dependencies between the beampatterns even from moderately distant points in the deformation cycle and hence support the notion of dynamic sensory encoding.

GPLEXUS: enabling genome-scale gene association network reconstruction and analysis for very large-scale expression data

The accurate construction and interpretation of gene association networks (GANs) is challenging, but crucial, to the understanding of gene function, interaction and cellular behavior at the genome level. Most current state-of-the-art computational methods for genome-wide GAN reconstruction require high-performance computational resources. However, even high-performance computing cannot fully address the complexity involved with constructing GANs from very large-scale expression profile datasets, especially for the organisms with medium to large size of genomes, such as those of most plant species. Here, we present a new approach, GPLEXUS (http://plantgrn.noble.org/GPLEXUS/), which integrates a series of novel algorithms in a parallel-computing environment to construct and analyze genome-wide GANs. GPLEXUS adopts an ultra-fast estimation for pairwise mutual information computing that is similar in accuracy and sensitivity to the Algorithm for the Reconstruction of Accurate Cellular Networks (ARACNE) method and runs ∼1000 times faster. GPLEXUS integrates Markov Clustering Algorithm to effectively identify functional subnetworks. Furthermore, GPLEXUS includes a novel ‘condition-removing’ method to identify the major experimental conditions in which each subnetwork operates from very large-scale gene expression datasets across several experimental conditions, which allows users to annotate the various subnetworks with experiment-specific conditions. We demonstrate GPLEXUS’s capabilities by construing global GANs and analyzing subnetworks related to defense against biotic and abiotic stress, cell cycle growth and division in Arabidopsis thaliana.

Entropy and the species accumulation curve: a novel entropy estimator via discovery rates of new species

Summary Estimating Shannon entropy and its exponential from incomplete samples is a central objective of many research fields. However, empirical estimates of Shannon entropy and its exponential depend strongly on sample size and typically exhibit substantial bias. This work uses a novel method to obtain an accurate, low‐bias analytic estimator of entropy, based on species frequency counts. Our estimator does not require prior knowledge of the number of species. We show that there is a close relationship between Shannon entropy and the species accumulation curve, which depicts the cumulative number of observed species as a function of sample size. We reformulate entropy in terms of the expected discovery rates of new species with respect to sample size, that is, the successive slopes of the species accumulation curve. Our estimator is obtained by applying slope estimators derived from an improved Good‐Turing frequency formula. Our method is also applied to estimate mutual information. Extensive simulations from theoretical models and real surveys show that if sample size is not unreasonably small, the resulting entropy estimator is nearly unbiased. Our estimator generally outperforms previous methods in terms of bias and accuracy (low mean squared error) especially when species richness is large and there is a large fraction of undetected species in samples. We discuss the extension of our approach to estimate Shannon entropy for multiple incidence data. The use of our estimator in constructing an integrated rarefaction and extrapolation curve of entropy (or mutual information) as a function of sample size or sample coverage (an aspect of sample completeness) is also discussed.

Estimation Bias in Maximum Entropy Models

Maximum entropy models have become popular statistical models in neuroscience and other areas in biology and can be useful tools for obtaining estimates of mutual information in biological systems. However, maximum entropy models fit to small data sets can be subject to sampling bias; i.e., the true entropy of the data can be severely underestimated. Here, we study the sampling properties of estimates of the entropy obtained from maximum entropy models. We focus on pairwise binary models, which are used extensively to model neural population activity. We show that if the data is well described by a pairwise model, the bias is equal to the number of parameters divided by twice the number of observations. If, however, the higher order correlations in the data deviate from those predicted by the model, the bias can be larger. Using a phenomenological model of neural population recordings, we find that this additional bias is highest for small firing probabilities, strong correlations and large population sizes—for the parameters we tested, a factor of about four higher. We derive guidelines for how long a neurophysiological experiment needs to be in order to ensure that the bias is less than a specified criterion. Finally, we show how a modified plug-in estimate of the entropy can be used for bias correction.

Bayesian and Quasi-Bayesian Estimators for Mutual Information from Discrete Data

Mutual information (MI) quantifies the statistical dependency between a pair of random variables, and plays a central role in the analysis of engineering and biological systems. Estimation of MI is difficult due to its dependence on an entire joint distribution, which is difficult to estimate from samples. Here we discuss several regularized estimators for MI that employ priors based on the Dirichlet distribution. First, we discuss three “quasi-Bayesian” estimators that result from linear combinations of Bayesian estimates for conditional and marginal entropies. We show that these estimators are not in fact Bayesian, and do not arise from a well-defined posterior distribution and may in fact be negative. Second, we show that a fully Bayesian MI estimator proposed by Hutter (2002), which relies on a fixed Dirichlet prior, exhibits strong prior dependence and has large bias for small datasets. Third, we formulate a novel Bayesian estimator using a mixture-of-Dirichlets prior, with mixing weights designed to produce an approximately flat prior over MI. We examine the performance of these estimators with a variety of simulated datasets and show that, surprisingly, quasi-Bayesian estimators generally outperform our Bayesian estimator. We discuss outstanding challenges for MI estimation and suggest promising avenues for future research.

An Estimate of Mutual Information that Permits Closed-Form Optimisation

We introduce a new estimate of mutual information between a dataset and a target variable that can be maximised analytically and has broad applicability in the field of machine learning and statistical pattern recognition. This estimate has previously been employed implicitly as an approximation to quadratic mutual information. In this paper we will study the properties of these estimates of mutual information in more detail, and provide a derivation from a perspective of pairwise interactions. From this perspective, we will show a connection between our proposed estimate and Laplacian eigenmaps, which so far has not been shown to be related to mutual information. Compared with other popular measures of mutual information, which can only be maximised through an iterative process, ours can be maximised much more efficiently and reliably via closed-form eigendecomposition.

A new histogram-based estimation technique of entropy and mutual information using mean squared error minimization

Mutual Information (MI) has extensively been used as a measure of similarity or dependence between random variables (or parameters) in different signal and image processing applications. However, MI estimation techniques are known to exhibit a large bias, a high Mean Squared Error (MSE), and can computationally be very costly. In order to overcome these drawbacks, we propose here a novel fast and low MSE histogram-based estimation technique for the computation of entropy and the mutual information. By minimizing the MSE, the estimation avoids the error accumulation problem of traditional methods. We derive an expression for the optimal number of bins to estimate the MI for both continuous and discrete random variables. Experimental results from a speech recognition problem and a computer aided diagnosis problem show the power of the proposed approach in estimating the optimal number of selected features with enhanced classification results compared to existing approaches.

On classification improvement by using an approximate discriminative hidden Markov model

HMMs are statistical models used in a very successful and effective form in speech recognition. However, HMM is a general model to describe the dynamic of stochastic processes; therefore it can be applied to a huge variety of biomedical signals. Usually, the HMM parameters are estimated by means of MLE (Maximum Likelihood Estimation) criterion. Nevertheless, MLE has as disadvantage that the distribution it is wanted to adjust is the distribution of each class, besides the models and/or data of other classes do not participate in the parameter re-estimation, as a result, the ML criterion is not directly related to reduce the error rate; it has led to many researchers to choice other training techniques known as discriminative training, including maximum mutual information (MMI) estimation. In this work, we carry out an EEG classification in order to compare HMM trained with both ML estimation and MMI estimation. The obtained results show a better performance in all database used.

Performance of Mutual Information Inference Methods Under Unknown Interference

In this paper, the problem of fast point-to-point multiple-input-multiple-output channel mutual information estimation is addressed, in the situation where the receiver undergoes unknown colored interference, whereas the channel with the transmitter is perfectly known. The considered scenario assumes that the estimation is based on a few channel use observations during a short sensing period. Using large dimensional random matrix theory, an estimator referred to as G-estimator is derived. This estimator is proved to be consistent as the number of antennas and observations grow large and its asymptotic performance is analyzed. In particular, the G-estimator satisfies a central limit theorem with asymptotic Gaussian fluctuations. Simulations are provided which strongly support the theoretical results, even for small system dimensions.

Scheduling of the Activations in Iterative Detection, Decoding, and Channel Estimation for MIMO-OFDM

An iterative receiver for a multiple-input-multiple-output (MIMO) orthogonal frequency-division multiplexing (OFDM) system is considered to jointly decode the transmitted bits and estimate the channel state. The receiver consists of the a posteriori probability (APP) algorithm, the repeat-accumulate (RA) decoder, and the least-squares (LS) channel estimator. An obvious problem, with more than two blocks in an iterative receiver, is to find the optimal activation schedule of the different blocks. This paper proposes to use extrinsic information transfer (EXIT) charts to characterize the behavior of the receiver blocks and find out the optimal activation schedule for them. A semi-analytical expression of the EXIT function is derived for the decision directed LS channel estimator. An algorithm is proposed to generate the EXIT function of the APP algorithm as a function of channel estimate mutual information (MI). Surface fitting is used to get closed form expressions for the EXIT functions of the APP algorithm and the RA decoder. Trellis search based algorithms are shown to find the convergence with the lowest possible complexity using the EXIT charts. With the proposed concept, the activation scheduling can be adapted to prevailing channel circumstances and unnecessary iterations will be avoided.

An Information Theoretic Approach to Constructing Machine Learning Criteria

Selecting a learning criterion is a constituent part of a machine learning problem statement requiring both accounting its adequacy to the data available and practical suitability of implementation. The paper presents an approach to the machine learning in accordance to information-theoretic criteria that are derived basing on the Rényi entropy of an arbitrary order. Meanwhile, a parameterized description of the machine learning is utilized combined with a corresponding technique of estimation of mutual information constructed basing on the Rényi entropies. This leads, finally, to a problem of the finite dimensional optimization to be solved by a suitable technique. The consideration proposed is preceded by a thorough review of existing information theoretic and entropy based approaches to the machine learning. The paper has been supported by a grant of the Russian Foundation for Basic Researches (RFBR): project 12-08-01205-a.