Non-parametric Mutual Information Research Articles

Ensemble Estimation of Generalized Mutual Information With Applications to Genomics

Mutual information is a measure of the dependence between random variables that has been used successfully in myriad applications in many fields. Generalized mutual information measures that go beyond classical Shannon mutual information have also received much interest in these applications. We derive the mean squared error convergence rates of kernel density-based plug-in estimators of general mutual information measures between two multidimensional random variables $\mathbf{X}$ and $\mathbf{Y}$ for two cases: 1) $\mathbf{X}$ and $\mathbf{Y}$ are continuous; 2) $\mathbf{X}$ and $\mathbf{Y}$ may have any mixture of discrete and continuous components. Using the derived rates, we propose an ensemble estimator of these information measures called GENIE by taking a weighted sum of the plug-in estimators with varied bandwidths. The resulting ensemble estimators achieve the $1/N$ parametric mean squared error convergence rate when the conditional densities of the continuous variables are sufficiently smooth. To the best of our knowledge, this is the first nonparametric mutual information estimator known to achieve the parametric convergence rate for the mixture case, which frequently arises in applications (e.g. variable selection in classification). The estimator is simple to implement and it uses the solution to an offline convex optimization problem and simple plug-in estimators. A central limit theorem is also derived for the ensemble estimators and minimax rates are derived for the continuous case. We demonstrate the ensemble estimator for the mixed case on simulated data and apply the proposed estimator to analyze gene relationships in single cell data.

Mutual information based feature subset selection in multivariate time series classification

This paper deals with supervised classification of multivariate time series. In particular, the goal is to propose a filter method to select a subset of time series. Consequently, we adopt the framework proposed by Brown al. [1]. The key point in this framework is the computation of the mutual information between the features, which allows us to measure the relevance of each feature subset. In our case, where the features are a time series, we use an adaptation of existing nonparametric mutual information estimators based on the k-nearest neighbor. Specifically, for the purpose of bringing these methods to the time series scenario, we rely on the use of dynamic time warping dissimilarity. Our experimental results show that our method is able to strongly reduce the number of time series while keeping or increasing the classification accuracy.

Graph embedded nonparametric mutual information for supervised dimensionality reduction.

In this paper, we propose a novel algorithm for dimensionality reduction that uses as a criterion the mutual information (MI) between the transformed data and their corresponding class labels. The MI is a powerful criterion that can be used as a proxy to the Bayes error rate. Furthermore, recent quadratic nonparametric implementations of MI are computationally efficient and do not require any prior assumptions about the class densities. We show that the quadratic nonparametric MI can be formulated as a kernel objective in the graph embedding framework. Moreover, we propose its linear equivalent as a novel linear dimensionality reduction algorithm. The derived methods are compared against the state-of-the-art dimensionality reduction algorithms with various classifiers and on various benchmark and real-life datasets. The experimental results show that nonparametric MI as an optimization objective for dimensionality reduction gives comparable and in most of the cases better results compared with other dimensionality reduction methods.

Dimensionality reduction based on non-parametric mutual information

In this paper we introduce a supervised linear dimensionality reduction algorithm which finds a projected input space that maximizes the mutual information between input and output values. The algorithm utilizes the recently introduced MeanNN estimator for differential entropy. We show that the estimator is an appropriate tool for the dimensionality reduction task. Next we provide a nonlinear regression algorithm based on the proposed dimensionality reduction approach. The regression algorithm achieves comparable to state-of-the-art performance on the standard data sets but is three orders of magnitude faster. In addition we describe applications of the proposed dimensionality reduction algorithm to reduced-complexity supervised and semisupervised classification tasks.

A maximum mutual information approach for constructing a 1D continuous control signal at a self-paced brain–computer interface

This paper addresses an important issue in a self-paced brain–computer interface (BCI): constructing subject-specific continuous control signal. To this end, we propose an alternative to the conventional regression/classification-based mechanism for building the transformation from EEG features into a univariate control signal. Based on information theory, the mechanism formulates the optimum transformation as maximizing the mutual information between the control signal and the mental state. We introduce a non-parametric mutual information estimate for general output distribution, and then develop a gradient-based algorithm to optimize the transformation using training data. We conduct an offline simulation study using motor imagery data from the BCI Competition IV Data Set I. The results show that the learning algorithm converged quickly, and the proposed method yielded significantly higher BCI performance than the conventional mechanism.