Distribution Of Mutual Information Research Articles

A smoothed bootstrap test for independence based on mutual information

A test for independence of multivariate time series based on the mutual information measure is proposed. First of all, a test for independence between two variables based on i.i.d. (time-independent) data is constructed and is then extended to incorporate higher dimensions and strictly stationary time series data. The smoothed bootstrap method is used to estimate the null distribution of mutual information. The experimental results reveal that the proposed smoothed bootstrap test performs better than the existing tests and can achieve high powers even for moderate dependence structures. Finally, the proposed test is applied to assess the actual independence of components obtained from independent component analysis (ICA).

Feature selection for gene function prediction using multi-labelled lazy learning

In multi-label learning, each instance in the training set is associated with a set of labels, and the task is to output a label set whose size is unknown a priori for each unseen instance. In this paper, feature selection for the multi-label method was proposed based on mutual information. In detail, we use the distribution of mutual information for feature selection in the multi-label problems. Our experiment was preceded on a multi-label lazy learning approach named ML-kNN, which is derived from the traditional k-Nearest Neighbour (KNN) algorithm. Experimental results on a real-world multi-label bioinformatics data show that ML-kNN with feature selection greatly outperforms the prior ML-kNN algorithm.

On the Mutual Information Distribution of OFDM-Based Spatial Multiplexing: Exact Variance and Outage Approximation

This paper considers the distribution of the mutual information of frequency-selective spatially-uncorrelated Rayleigh fading MIMO channels. Results are presented for OFDM-based spatial multiplexing. New exact closed-form expressions are derived for the variance of the mutual information. In contrast to previous results, our new expressions apply for systems with both arbitrary numbers of antennas and arbitrary-length channels. Simplified expressions are also presented for high and low SNR regimes. The analytical variance results are used to provide accurate analytical approximations for the distribution of the mutual information and the outage capacity.

Multiple-Antenna Channel Hardening and Its Implications for Rate Feedback and Scheduling

Wireless data traffic is expected to grow over the next few years and the technologies that will provide data services are still being debated. One possibility is to use multiple antennas at base stations and terminals to get very high spectral efficiencies in rich scattering environments. Such multiple-input/multiple-output (MIMO) channels can then be used in conjunction with scheduling and rate-feedback algorithms to further increase channel throughput. This paper provides an analysis of the expected gains due to scheduling and bits needed for rate feedback. Our analysis requires an accurate approximation of the distribution of the MIMO channel mutual information. Because the exact distribution of the mutual information in a Rayleigh-fading environment is difficult to analyze, we prove a central limit theorem for MIMO channels with a large number of antennas. While the growth in average mutual information (capacity) of a MIMO channel with the number of antennas is well understood, it turns out that the variance of the mutual information can grow very slowly or even shrink as the number of antennas grows. We discuss implications of this "channel-hardening" result for data and voice services, scheduling, and rate feedback. We also briefly discuss the implications when shadow fading effects are included.

Distribution of mutual information from complete and incomplete data

Mutual information is widely used, in a descriptive way, to measure the stochastic dependence of categorical random variables. In order to address questions such as the reliability of the descriptive value, one must consider sample-to-population inferential approaches. This paper deals with the posterior distribution of mutual information, as obtained in a Bayesian framework by a second-order Dirichlet prior distribution. The exact analytical expression for the mean, and analytical approximations for the variance, skewness and kurtosis are derived. These approximations have a guaranteed accuracy level of the order O( n −3), where n is the sample size. Leading order approximations for the mean and the variance are derived in the case of incomplete samples. The derived analytical expressions allow the distribution of mutual information to be approximated reliably and quickly. In fact, the derived expressions can be computed with the same order of complexity needed for descriptive mutual information. This makes the distribution of mutual information become a concrete alternative to descriptive mutual information in many applications which would benefit from moving to the inductive side. Some of these prospective applications are discussed, and one of them, namely feature selection, is shown to perform significantly better when inductive mutual information is used.

Distribution of mutual information

In the analysis of time series from nonlinear sources, mutual information (MI) is used as a nonlinear statistical criterion for the selection of an appropriate time delay in time delay reconstruction of the state space. MI is a statistic over the sets of sequences associated with the dynamical source, and we examine here the distribution of MI, thus going beyond the familiar analysis of its average alone. We give for the first time the distribution of MI for a standard, classical communications channel with Gaussian, additive white noise. For time series analysis of a dynamical system, we show how to determine the distribution of MI and discuss the implications for the use of average mutual information (AMI) in selecting time delays in phase space reconstruction.