Mutual Information Estimation Research Articles

Improved Mutual Information Estimation

We propose to estimate the KL divergence using a relaxed likelihood ratio estimation in a Reproducing Kernel Hilbert space. We show that the dual of our ratio estimator for KL in the particular case of Mutual Information estimation corresponds to a lower bound on the MI that is related to the so called Donsker Varadhan lower bound. In this dual form, MI is estimated via learning a witness function discriminating between the joint density and the product of marginal, as well as an auxiliary scalar variable that enforces a normalization constraint on the likelihood ratio. By extending the function space to neural networks, we propose an efficient neural MI estimator, and validate its performance on synthetic examples, showing advantage over the existing baselines. We demonstrate its strength in large-scale self-supervised representation learning through MI maximization.

Adversarial balancing-based representation learning for causal effect inference with observational data

Learning causal effects from observational data greatly benefits a variety of domains such as health care, education, and sociology. For instance, one could estimate the impact of a new drug on specific individuals to assist clinical planning and improve the survival rate. In this paper, we focus on studying the problem of estimating the Conditional Average Treatment Effect (CATE) from observational data. The challenges for this problem are two-fold: on the one hand, we have to derive a causal estimator to estimate the causal quantity from observational data, in the presence of confounding bias; on the other hand, we have to deal with the identification of the CATE when the distributions of covariates over the treatment group units and the control units are imbalanced. To overcome these challenges, we propose a neural network framework called Adversarial Balancing-based representation learning for Causal Effect Inference (ABCEI), based on recent advances in representation learning. To ensure the identification of the CATE, ABCEI uses adversarial learning to balance the distributions of covariates in the treatment and the control group in the latent representation space, without any assumptions on the form of the treatment selection/assignment function. In addition, during the representation learning and balancing process, highly predictive information from the original covariate space might be lost. ABCEI can tackle this information loss problem by preserving useful information for predicting causal effects under the regularization of a mutual information estimator. The experimental results show that ABCEI is robust against treatment selection bias, and matches/outperforms the state-of-the-art approaches. Our experiments show promising results on several datasets, encompassing several health care (and other) domains.

Mutual information matrix based on asymmetric Shannon entropy for nonlinear interactions of time series

Zhao et al. (Nonlin. Dyn. 88, 477-487, 2017) presented the mutual information matrix (MIM) analysis for the study of nonlinear interactions in multivariate time series as an extension of Random Matrix Theory analysis. They considered the histogram estimation of mutual information based on Shannon entropy for discrete distributions. This paper is motivated by the latter, extending MIM analysis from a nonparametric and probabilistic discrete approach to a parametric and probabilistic continuous approach. Specifically, this paper presents the MIM based on Maximum Likelihood Estimators (MLEs) for flexible and tractable families of continuous multivariate distributions, called multivariate skew-elliptical families of distributions. This method focus on multivariate skew-Gaussian and skew-t distributions that allow modeling skewness and heavy-tails, respectively. Performance of the proposed methodology is illustrated by numerical results given by sinusoidal and vector autoregressive fractionally integrated moving-average models, and applied to a meteorological monitoring network data set. Results show that the consideration of skewness and heavy-tails in the transformed ozone time series produced some differences in the MIM estimations compared with those obtained by applying histogram estimations to transformed data. Given that mutual information index (MII) increases in line with the number of bins for the histogram estimator, the proposed methodology based on MLEs considered more robust estimators with respect to the histogram ones to determine the MII of multivariate time series.

Cross-Scale Causality and Information Transfer in Simulated Epileptic Seizures.

An information-theoretic approach for detecting causality and information transfer was applied to phases and amplitudes of oscillatory components related to different time scales and obtained using the wavelet transform from a time series generated by the Epileptor model. Three main time scales and their causal interactions were identified in the simulated epileptic seizures, in agreement with the interactions of the model variables. An approach consisting of wavelet transform, conditional mutual information estimation, and surrogate data testing applied to a single time series generated by the model was demonstrated to be successful in the identification of all directional (causal) interactions between the three different time scales described in the model. Thus, the methodology was prepared for the identification of causal cross-frequency phase–phase and phase–amplitude interactions in experimental and clinical neural data.

Inferring functional connectivity through graphical directed information

Objective. Accurate inference of functional connectivity is critical for understanding brain function. Previous methods have limited ability distinguishing between direct and indirect connections because of inadequate scaling with dimensionality. This poor scaling performance reduces the number of nodes that can be included in conditioning. Our goal was to provide a technique that scales better and thereby enables minimization of indirect connections. Approach. Our major contribution is a powerful model-free framework, graphical directed information (GDI), that enables pairwise directed functional connections to be conditioned on the activity of substantially more nodes in a network, producing a more accurate graph of functional connectivity that reduces indirect connections. The key technology enabling this advancement is a recent advance in the estimation of mutual information (MI), which relies on multilayer perceptrons and exploiting an alternative representation of the Kullback–Leibler divergence definition of MI. Our second major contribution is the application of this technique to both discretely valued and continuously valued time series. Main results. GDI correctly inferred the circuitry of arbitrary Gaussian, nonlinear, and conductance-based networks. Furthermore, GDI inferred many of the connections of a model of a central pattern generator circuit in Aplysia, while also reducing many indirect connections. Significance. GDI is a general and model-free technique that can be used on a variety of scales and data types to provide accurate direct connectivity graphs and addresses the critical issue of indirect connections in neural data analysis.

Addressing indirect frequency coupling via partial generalized coherence

Distinguishing between direct and indirect frequency coupling is an important aspect of functional connectivity analyses because this distinction can determine if two brain regions are directly connected. Although partial coherence quantifies partial frequency coupling in the linear Gaussian case, we introduce a general framework that can address even the nonlinear and non-Gaussian case. Our technique, partial generalized coherence (PGC), expands prior work by allowing pairwise frequency coupling analyses to be conditioned on other processes, enabling model-free partial frequency coupling results. By taking advantage of recent advances in conditional mutual information estimation, we are able to implement our technique in a way that scales well with dimensionality, making it possible to condition on many processes and produce a partial frequency coupling graph. We analyzed both linear Gaussian and nonlinear simulated networks. We then performed PGC analysis of calcium recordings from mouse olfactory bulb glomeruli under anesthesia and quantified the dominant influence of breathing-related activity on the pairwise relationships between glomeruli for breathing-related frequencies. Overall, we introduce a technique capable of eliminating indirect frequency coupling in a model-free way, empowering future research to correct for potentially misleading frequency interactions in functional connectivity analyses.

A method for continuous-range sequence analysis with Jensen-Shannon divergence

Mutual Information (MI) is a useful Information Theory tool for the recognition of mutual dependence between data sets. Several methods have been developed fore estimation of MI when both data sets are of the discrete type or when both are of the continuous type. However, MI estimation between a discrete range data set and a continuous range data set has not received so much attention. We therefore present here a method for the estimation of MI for this case, based on the kernel density approximation. This calculation may be of interest in diverse contexts. Since MI is closely related to the Jensen Shannon divergence, the method developed here is of particular interest in the problems of sequence segmentation and set comparisons.

The influence of environmental factors on the intensity of typhoon Mirinae in the South China Sea and its forecast analysis

The rapidly intensifying typhoon Mirinae (1603) originated in the South China Sea (SCS) has been selected to analyze the high and low level circulations, vertical wind shear (VWS) and ocean conditions in and around the SCS when Mirinae was strengthening. Further, the fuzzy neural network (FNN) ensemble model has been developed for 24-hour intensity prediction based on climatology and persistence factors and dynamic factors of numerical prediction products as potential predictors. The underlying conditions were favorable for Mirinae, however, it did not be intensified to a strong typhoon because of the weak high and low level circulations and strong VWS. For the FNN ensemble prediction model input, predictors were selected by employing both a novel stepwise variable selection algorithm based on the k-nearest mutual information estimation and Laplacian Eigenmaps of manifold learning, and then the main prediction information hidden in high dimensional data set was extracted with low-dimensional structure. Results show that, the predictions of the new model are stable and reliable, and the forecast accuracy is high, providing a new forecasting tool and modeling method for the objective forecast of typhoon intensity.

Characterization and Exploitation of Quasi Determinism in Multi-Conductor Power Line Communication Noise

Several experiments and field trials have shown that deterministic components characterize the Power Line Communication (PLC) noise across multiple-conductors. This article aims at understanding the main origins of these noise components and evaluate several methods to characterize them statistically. The correlation, distance-correlation, mutual information estimation, and a proposed linear coefficient ratio, are used to analyze the noise time-series statistically. The analysis of the PLC noise deterministic components allow us to develop an enhanced noise model that includes them. The presence of deterministic noise fosters the development of coding and decoding algorithms that aim at mitigating the noise effect. New decoding techniques called Quasi-Deterministic-Decoding are detailed and tested with single-input-multiple-output (SIMO) 1×2 and multiple-input-multiple-output (MIMO) 2×2 transmission configurations using real PLC noise and channel realizations. The proposed model leads to a better representation of the PLC noise traces and, consequently, the development of improved coding algorithms for reliable data transmission at low signal-to-noise ratios.

Multimodal Representation Learning via Maximization of Local Mutual Information.

We propose and demonstrate a representation learning approach by maximizing the mutual information between local features of images and text. The goal of this approach is to learn useful image representations by taking advantage of the rich information contained in the free text that describes the findings in the image. Our method trains image and text encoders by encouraging the resulting representations to exhibit high local mutual information. We make use of recent advances in mutual information estimation with neural network discriminators. We argue that the sum of local mutual information is typically a lower bound on the global mutual information. Our experimental results in the downstream image classification tasks demonstrate the advantages of using local features for image-text representation learning. Our code is available at: https://github.com/RayRuizhiLiao/mutual_info_img_txt.

SSSpaNG! Stellar Spectra as Sparse, data-driven, Non-Gaussian processes

Abstract Upcoming million-star spectroscopic surveys have the potential to revolutionize our view of the formation and chemical evolution of the Milky Way. Realizing this potential requires automated approaches to optimize estimates of stellar properties, such as chemical element abundances, from the spectra. The sheer volume and quality of the observations strongly motivate that these approaches should be driven by the data. With this in mind, we introduce SSSpaNG: a data-driven non-Gaussian Process model of stellar spectra. We demonstrate the capabilities of SSSpaNG using a sample of APOGEE red clump stars, whose model parameters we infer using Gibbs sampling. By pooling information between stars to infer their covariance, we permit clear identification of the correlations between spectral pixels. Harnessing this correlation structure, we infer the true spectrum of each red clump star, inpainting missing regions and denoising by a factor of at least two for stars with signal-to-noise of ∼20. As we marginalize over the covariance matrix of the spectra, the effective prior on these true spectra is non-Gaussian and sparsifying, favouring typically small but occasionally large excursions from the mean. The high-fidelity inferred spectra produced with our approach will enable improved chemical elemental abundance estimates for individual stars. Our model also allows us to quantify the information gained by observing portions of a star’s spectrum, and thereby define the most mutually informative spectral regions. Using 25 windows centred on elemental absorption lines, we demonstrate that the iron-peak and alpha-process elements are particularly mutually informative for these spectra, and that the majority of information about a target window is contained in the 10-or-so most informative windows. Such mutual information estimates have the potential to inform models of nucleosynthetic yields and the design of future observations. Our code is made publicly available at https://github.com/sfeeney/ddspectra.

Application of mutual information estimation for predicting the structural stability of pentapeptides

AbstractA novel non-parametric method for mutual information estimation is presented. The method is suited for informative feature selection in classification and regression problems. Performance of the method is demonstrated on problem of stable short peptide classification.

Recognizing Proteins with Binding Function in Elymus nutans Based on Machine Learning Methods.

We research the binding function proteins in Elymus nutans. Recognition for proteins is essential for study of biology. Machine learning methods have been widely used for the prediction of proteins. We used BLAST software for the function annotations of Elymus nutans. Besides, we used machine learning methods to recognize proteins which are not annotated by the software. In the process, we focused on identifying the proteins with binding functions. In our research, features are extracted by four algorithms, and then selected by mutual information estimator. Here three classifiers are constructed based on K-nearest neighbour algorithm and gradient boosting algorithm. Experimental results show that there are 848 proteins with ATP binding function, 113 proteins with heme binding function, 315 proteins with zinc-ion binding function, 135 proteins with GTP binding function and 21 proteins with ADP binding function. Furthermore, we have successfully predicted the functions of 10 special protein sequences whose function annotations cannot be obtained by making sequence alignment with seven famous protein databases. Among them, seven sequences have ATP binding functions, one sequence has heme binding function, one sequence has zinc-ion binding function and the other one has GTP binding function.

Conditional Mutual Information Estimation for Mixed, Discrete and Continuous Data

Fields like public health, public policy, and social science often want to quantify the degree of dependence between variables whose relationships take on unknown functional forms. Typically, in fact, researchers in these fields are attempting to evaluate causal theories, and so want to quantify dependence after conditioning on other variables that might explain, mediate or confound causal relations. One reason conditional mutual information is not more widely used for these tasks is the lack of estimators which can handle combinations of continuous and discrete random variables, common in applications. This article develops a new method for estimating mutual and conditional mutual information for data samples containing a mix of discrete and continuous variables. We prove that this estimator is consistent and show, via simulation, that it is more accurate than similar estimators.

Spectral Analysis and Mutual Information Estimation of Left and Right Intracardiac Electrograms during Ventricular Fibrillation.

Ventricular fibrillation (VF) signals are characterized by highly volatile and erratic electrical impulses, the analysis of which is difficult given the complex behavior of the heart rhythms in the left (LV) and right ventricles (RV), as sometimes shown in intracardiac recorded Electrograms (EGM). However, there are few studies that analyze VF in humans according to the simultaneous behavior of heart signals in the two ventricles. The objective of this work was to perform a spectral and a non-linear analysis of the recordings of 22 patients with Congestive Heart Failure (CHF) and clinical indication for a cardiac resynchronization device, simultaneously obtained in LV and RV during induced VF in patients with a Biventricular Implantable Cardioverter Defibrillator (BICD) Contak Renewal IVTM (Boston Sci.). The Fourier Transform was used to identify the spectral content of the first six seconds of signals recorded in the RV and LV simultaneously. In addition, measurements that were based on Information Theory were scrutinized, including Entropy and Mutual Information. The results showed that in most patients the spectral envelopes of the EGM sources of RV and LV were complex, different, and with several frequency peaks. In addition, the Dominant Frequency (DF) in the LV was higher than in the RV, while the Organization Index (OI) had the opposite trend. The entropy measurements were more regular in the RV than in the LV, thus supporting the spectral findings. We can conclude that basic stochastic processing techniques should be scrutinized with caution and from basic to elaborated techniques, but they can provide us with useful information on the biosignals from both ventricles during VF.

Label-free discrimination and quantitative analysis of oxidative stress induced cytotoxicity and potential protection of antioxidants using Raman micro-spectroscopy and machine learning

Diesel exhaust particles (DEPs) are major constituents of air pollution and associated with numerous oxidative stress-induced human diseases. In vitro toxicity studies are useful for developing a better understanding of species-specific in vivo conditions. Conventional in vitro assessments based on oxidative biomarkers are destructive and inefficient. In this study, Raman spectroscopy, as a non-invasive imaging tool, was used to capture the molecular fingerprints of overall cellular component responses (nucleic acid, lipids, proteins, carbohydrates) to DEP damage and antioxidant protection. We apply a novel data visualization algorithm called PHATE, which preserves both global and local structure, to display the progression of cell damage over DEP exposure time. Meanwhile, a mutual information (MI) estimator was used to identify the most informative Raman peaks associated with cytotoxicity. A health index was defined to quantitatively assess the protective effects of two antioxidants (resveratrol and mesobiliverdin IXα) against DEP induced cytotoxicity. In addition, a number of machine learning classifiers were applied to successfully discriminate different treatment groups with high accuracy. Correlations between Raman spectra and immunomodulatory cytokine and chemokine levels were evaluated. In conclusion, the combination of label-free, non-disruptive Raman micro-spectroscopy and machine learning analysis is demonstrated as a useful tool in quantitative analysis of oxidative stress induced cytotoxicity and for effectively assessing various antioxidant treatments, suggesting that this framework can serve as a high throughput platform for screening various potential antioxidants based on their effectiveness at battling the effects of air pollution on human health.

Mutual information based feature subset selection in multivariate time series classification

This paper deals with supervised classification of multivariate time series. In particular, the goal is to propose a filter method to select a subset of time series. Consequently, we adopt the framework proposed by Brown al. [1]. The key point in this framework is the computation of the mutual information between the features, which allows us to measure the relevance of each feature subset. In our case, where the features are a time series, we use an adaptation of existing nonparametric mutual information estimators based on the k-nearest neighbor. Specifically, for the purpose of bringing these methods to the time series scenario, we rely on the use of dynamic time warping dissimilarity. Our experimental results show that our method is able to strongly reduce the number of time series while keeping or increasing the classification accuracy.

Statistical Estimation of Mutual Information for Mixed Model

Asymptotic unbiasedness and L2-consistency are established for various statistical estimates of mutual information in the mixed models framework. Such models are important, e.g., for analysis of medical and biological data. The study of the conditional Shannon entropy as well as new results devoted to statistical estimation of the differential Shannon entropy are employed essentially. Theoretical results are completed by computer simulations for logistic regression model with different parameters. The numerical experiments demonstrate that new statistics, proposed by the authors, have certain advantages.

Preserving physically important variables in optimal event selections: a case study in Higgs physics

Analyses of collider data, often assisted by modern Machine Learning methods, condense a number of observables into a few powerful discriminants for the separation of the targeted signal process from the contributing backgrounds. These discriminants are highly correlated with important physical observables; using them in the event selection thus leads to the distortion of physically relevant distributions. We present a novel method based on a differentiable estimate of mutual information, a measure of non-linear dependency between variables, to construct a discriminant that is statistically independent of a number of selected observables, and so manages to preserve their distributions in the event selection. Our strategy is evaluated in a realistic setting, the analysis of the Standard Model Higgs boson decaying into a pair of bottom quarks. Using the distribution of the invariant mass of the di-b-jet system to extract the Higgs boson signal strength, our method achieves state-of-the-art performance compared to other decorrelation techniques, while significantly improving the sensitivity of a similar, cut-based, analysis published by ATLAS.

Convergence Behavior of DNNs with Mutual-Information-Based Regularization.

Information theory concepts are leveraged with the goal of better understanding and improving Deep Neural Networks (DNNs). The information plane of neural networks describes the behavior during training of the mutual information at various depths between input/output and hidden-layer variables. Previous analysis revealed that most of the training epochs are spent on compressing the input, in some networks where finiteness of the mutual information can be established. However, the estimation of mutual information is nontrivial for high-dimensional continuous random variables. Therefore, the computation of the mutual information for DNNs and its visualization on the information plane mostly focused on low-complexity fully connected networks. In fact, even the existence of the compression phase in complex DNNs has been questioned and viewed as an open problem. In this paper, we present the convergence of mutual information on the information plane for a high-dimensional VGG-16 Convolutional Neural Network (CNN) by resorting to Mutual Information Neural Estimation (MINE), thus confirming and extending the results obtained with low-dimensional fully connected networks. Furthermore, we demonstrate the benefits of regularizing a network, especially for a large number of training epochs, by adopting mutual information estimates as additional terms in the loss function characteristic of the network. Experimental results show that the regularization stabilizes the test accuracy and significantly reduces its variance.