Minimum Mutual Information Research Articles

Addendum to “On Universal Simulation of Information Sources Using Training Data”

In a recent paper (Merhav and Weinberger, IEEE Trans. Inf. Theory, vol.50, no.1, p.5-20, 2004) we studied the problem of universal simulation of an unknown information source of a certain parametric family, given a training sequence from that source and given a limited budget of purely random bits. The goal was to generate another random sequence (of the same length or shorter), whose probability law is identical to that of the given training sequence, but with minimum statistical dependency (minimum mutual information) between the input training sequence and the output sequence. In this addendum, we point out a concrete optimal simulation scheme that is easy to implement, as opposed to the nonconstructive existence result in that paper, and we make a number of additional observations on the universal simulation problem.

Prostate cancer multi-feature analysis using trans-rectal ultrasound images

This note focuses on extracting and analysing prostate texture features from trans-rectal ultrasound (TRUS) images for tissue characterization. One of the principal contributions of this investigation is the use of the information of the images' frequency domain features and spatial domain features to attain a more accurate diagnosis. Each image is divided into regions of interest (ROIs) by the Gabor multi-resolution analysis, a crucial stage, in which segmentation is achieved according to the frequency response of the image pixels. The pixels with a similar response to the same filter are grouped to form one ROI. Next, from each ROI two different statistical feature sets are constructed; the first set includes four grey level dependence matrix (GLDM) features and the second set consists of five grey level difference vector (GLDV) features. These constructed feature sets are then ranked by the mutual information feature selection (MIFS) algorithm. Here, the features that provide the maximum mutual information of each feature and class (cancerous and non-cancerous) and the minimum mutual information of the selected features are chosen, yeilding a reduced feature subset. The two constructed feature sets, GLDM and GLDV, as well as the reduced feature subset, are examined in terms of three different classifiers: the condensed k-nearest neighbour (CNN), the decision tree (DT) and the support vector machine (SVM). The accuracy classification results range from 87.5% to 93.75%, where the performance of the SVM and that of the DT are significantly better than the performance of the CNN.

Criteria based on mutual information minimization for blind source separation in post nonlinear mixtures

This work deals with the problem of blind source separation solved by minimization of mutual information. After having chosen a model for the mixture, we focus on two methods. One is based on the minimization of an estimation of I, the mutual information. The other one uses a minimization of an estimation of C, the mutual information after transforming all the joint entropy terms. We show the differences between these two approaches by studying statistical properties of the two estimators. In this paper, we derive the bias of the estimators of the two criteria I and C. It is shown that under the hypothesis of independence, the estimator of I is asymptotically unbiased even if the bandwidth is kept fixed, whereas with a fixed bandwidth the estimator of C is not asymptotically unbiased. Further, the minimization is achieved by a relative gradient descent method and we show the differences between criteria I and C through the expression of their relative gradients.

A general approach for mutual information minimization and its application to blind source separation

In this paper, a nonparametric “gradient” of the mutual information is first introduced. It is used for showing that mutual information has no local minima. Using the introduced “gradient”, two general gradient based approaches for minimizing mutual information in a parametric model are then presented. These approaches are quite general, and principally they can be used in any mutual information minimization problem. In blind source separation, these approaches provide powerful tools for separating any complicated (yet separable) mixing model. In this paper, they are used to develop algorithms for separating four separable mixing models: linear instantaneous, linear convolutive, post nonlinear (PNL) and convolutive post nonlinear (CPNL) mixtures.

Evolving Neural Network Ensembles by Minimization of Mutual Information

Learning and evolution are two fundamental forms of adaptation. There has been a great interest in combining learning and evolution with neural networks in recent years. This paper presents a hybrid learning system for learning and designing of neural network ensembles based on negative correlation learning and evolutionary learning. The idea of the hybrid learning system is to regard the population of neural networks as an ensemble, and the evolutionary process as the design of neural network ensembles. Two fitness sharing techniques have been used in the evolutionary process. One is based on the covering set. The other is to use the concept of mutual information. The effectiveness of such hybrid learning approach was tested on two real-world problems.

Independent component analysis via optimum combining of kurtosis and skewness-based criteria

This paper introduces blind separation methods that are based on minimization of mutual information. Direct minimization of mutual information leads to estimating functions that change on every iteration of separating algorithm unlike in the maximum likelihood approach employing fixed non-linearity. We propose objective functions for source separation that are comprised of a symmetric and an asymmetric part. This allows for separating signals that may have skewed distributions. The optimal weighting between the symmetric and the asymmetric part is determined from the data based on an efficacy measure. The performance of the proposed objective functions is studied in cases where some source signals may be asymmetrically distributed. The capability of adapting to different type of source distributions is demonstrated in simulations.

The Uniform Distribution as a Universal Prior

In this correspondence, we discuss the properties of the uniform prior as a universal prior, i.e., a prior that induces a mutual information that is simultaneously close to the capacity for all channels. We determine bounds on the amount of the mutual information loss in using the uniform prior instead of the capacity-achieving prior. Specifically, for the class of binary input channels with any output alphabet, we show that the Z-channel has the minimal mutual information with uniform prior, out of all channels with a given capacity. From this, we conclude that the degradation of the mutual information with respect to the capacity is at most 0.011 bit, and as was shown previously, at most 6%. A related result is that the capacity-achieving prior, for any channel, is not far from uniform. Some of these results are extended to channels with nonbinary input.

Bifurcation diagram, noise reduction and period-four cycle on low frequency current oscillations in a semi-insulating GaAs sample

Low frequency oscillations (LFO) on current I( V, t) curves were studied in a GaAs sample grown by molecular beam epitaxy. Our measurements were carried out at a temperature of 200 K and under infrared illumination. The nonlinear I( V) characteristics give rise to spontaneous oscillations under constant applied bias V. The attractors were constructed by the delay vector technique with the delay value given by minimal mutual information. A bifurcation diagram was constructed from the minima sequence as a function of the applied bias. A noise reduction algorithm applied to the raw data allowed the cleaning of the bifurcation diagram. We have identified a bifurcation route of periodicity following the sequence 0–1–2–4–2–∞ having the applied bias as the control parameter. Fourier transforms is a merit figure of the noise reduction procedure and also a qualitative proof of the chaotic regime.

Bifurcation diagram, noise reduction and period-four cycle on low frequency current oscillations in a semi-insulating GaAs sample

Low frequency oscillations (LFO) on current I ( V , t ) curves were studied in a GaAs sample grown by molecular beam epitaxy. Our measurements were carried out at a temperature of 200 K and under infrared illumination. The nonlinear I ( V ) characteristics give rise to spontaneous oscillations under constant applied bias V . The attractors were constructed by the delay vector technique with the delay value given by minimal mutual information. A bifurcation diagram was constructed from the minima sequence as a function of the applied bias. A noise reduction algorithm applied to the raw data allowed the cleaning of the bifurcation diagram. We have identified a bifurcation route of periodicity following the sequence 0–1–2–4–2–∞ having the applied bias as the control parameter. Fourier transforms is a merit figure of the noise reduction procedure and also a qualitative proof of the chaotic regime.

One-bit-matching conjecture for independent component analysis.

The one-bit-matching conjecture for independent component analysis (ICA) could be understood from different perspectives but is basically stated as "all the sources can be separated as long as there is a one-to-one same-sign-correspondence between the kurtosis signs of all source probability density functions (pdf's) and the kurtosis signs of all model pdf's" (Xu, Cheung, & Amari, 1998a). This conjecture has been widely believed in the ICA community and implicitly supported by many ICA studies, such as the Extended Infomax (Lee, Girolami, & Sejnowski, 1999) and the soft switching algorithm (Welling & Weber, 2001). However, there is no mathematical proof to confirm the conjecture theoretically. In this article, only skewness and kurtosis are considered, and such a mathematical proof is given under the assumption that the skewness of the model densities vanishes. Moreover, empirical experiments are demonstrated on the robustness of the conjecture as the vanishing skewness assumption breaks. As a by-product, we also show that the kurtosis maximization criterion (Moreau & Macchi, 1996) is actually a special case of the minimum mutual information criterion for ICA.

On Universal Simulation of Information Sources Using Training Data

We consider the problem of universal simulation of an unknown random process, or information source, of a certain parametric family, given a training sequence from that source and given a limited budget of purely random bits. The goal is to generate another random sequence (of the same length or shorter), whose probability law is identical to that of the given training sequence, but with minimum statistical dependency (minimum mutual information) between the input training sequence and the output sequence. We derive lower bounds on the mutual information that are shown to he achievable by conceptually simple algorithms proposed here. We show that the behavior of the minimum achievable mutual information depends critically on the relative amount of random bits and on the lengths of the input sequence and the output sequence. While in the ordinary (nonuniversal) simulation problem, the number of random bits per symbol must exceed the entropy rate H of the source in order to simulate it faithfully, in the universal simulation problem considered here, faithful preservation of the probability law is not a problem, yet the same minimum rate of H random bits per symbol is still needed to essentially eliminate the statistical dependency between the input sequence and the output sequence. The results are extended to more general information measures.

Construction of genomic networks using mutual-information clustering and reversible-jump Markov-chain-Monte-Carlo predictor design

We consider the problem of constructing gene regulatory networks from expression data using the probabilistic-Boolean-network paradigm. We propose to construct the networks according to the following stages. Firstly, we determine the number of possible parent gene sets and the input sets of gene variables corresponding to each gene, and this is done by using a novel clustering technique based on mutual information minimization. Simulated annealing is employed to solve the optimization problem. After such initial gene clustering, we restrict our attention to the class of different functions from the possible parent gene sets to each target gene. Secondly, each function is then modeled by a perceptron consisting of a linear term and a nonlinear term. A reversible-jump Markov-chain-Monte-Carlo (MCMC) technique is used to calculate the model order and the parameters. Finally, the coefficient of determination (CoD) is employed to compute the probability of selecting different predictors for each gene. To test this approach for constructing gene regulatory networks, we have carried out computational experiments using data from known gene response pathways including ionizing radiation and downstream targets of inactivating gene mutations.

Kernel-based topographic map formation achieved with an information-theoretic approach

A new information-theoretic learning algorithm is introduced for kernel-based topographic map formation. The kernels are allowed to overlap and move freely in the input space, and to have differing kernel ranges. We start with Linsker's infomax principle and observe that it cannot be readily extended to our case, exactly due to the presence of kernels. We then consider Bell and Sejnowski's generalization of Linsker's infomax principle, which suggests differential entropy maximization, and add a second component to be optimized, namely, mutual information minimization between the kernel outputs, in order to take into account the kernel overlap, and thus the topographic map's output redundancy. The result is joint entropy maximization of the kernel outputs, which we adopt as our learning criterion. We derive a learning algorithm and verify its performance both for a synthetic example, for which the optimal result can be derived analytically, and for a classic real-world example.

Parametric approach to blind deconvolution of nonlinear channels

A parametric procedure for blind inversion of nonlinear channels is proposed, based on a recent method of blind source separation in nonlinear mixtures. Two parametric models are developed: a polynomial model and a neural model. The method, based on the minimization of the output mutual information, needs the knowledge of log-derivative of the input distribution ( score function). Each algorithm consists of three adaptive blocks: one for estimating the score function, and two other blocks for estimating the inverses of the linear and nonlinear parts of the channel. Experiments show that the algorithms perform efficiently, even for hard nonlinear distortion.

Gambling for the mnemonically impaired

We obtain asymptotically tight bounds on the maximum amount of information that a single bit of memory can retain about the entire past. At each of n successive epochs, a single fair bit is generated and a one-bit memory is updated according to a family of memory update rules (possibly probabilistic and time-dependent) depending only on the value of the new input bit and on the current state of the memory. The problem is to estimate the supremum over all possible update rules of the minimum mutual information between the state of the memory at time (n + 1) and each of the previous n input bits. We show that this supremum is asymptotically equal to 1/(2n/sup 2/ ln 2) bit, as conjectured by Venkatesh and Franklin (1991). We use this result to derive asymptotically sharp estimates of related maximin correlations between the memory and the input bits, thus resolving two more questions left open by Venkatesh and Franklin and by Komlos et al. (1993). Finally, we generalize the results to the case of an m-bit memory, again obtaining asymptotically tight bounds in many cases.

Comparison of maximum entropy and minimal mutual information in a nonlinear setting

In blind source separation (BSS), two different separation techniques are mainly used: minimal mutual information (MMI), where minimization of the mutual output information yields an independent random vector, and maximum entropy (ME), where the output entropy is maximized. However, it is yet unclear why ME should solve the separation problem, i.e. result in an independent vector. Yang and Amari have given a partial confirmation for ME in the linear case in [18], where they prove that under the assumption of vanishing expectation of the sources ME does not change the solutions of MMI except for scaling and permutation. In this paper, we generalize Yang and Amari's approach to nonlinear BSS problems, where random vectors are mixed by output functions of layered neural networks. We show that certain solution points of MMI are kept fixed by ME if no scaling in all layers is allowed. In general, ME, however, might also change the scaling in the non-output network layers, hence, leaving the MMI solution points. Therefore, we conclude this paper by suggesting that in nonlinear ME algorithms, the norm of all weight matrix rows of each non-output layer should be kept fixed in later epochs during network training.

General multilayer perceptron demixer scheme for nonlinear blind signal separation

A new technique is presented for instantaneous blind signal separation from nonlinear mixtures using a general neural network based demixer scheme. The nonlinear demixer model follows directly from the general mixer model. In the first part of the paper a general mixer model is described which includes linear mixtures as a special case. In the second part the general framework for a demixer based on a feedforward multilayer perceptron (FMLP) employing a class of continuously differentiable nonlinear functions is presented. A detailed derivation of the learning algorithm used to adapt the demixer's parameters is given. Cost functions based on both maximum entropy (ME) and minimum mutual information (MMI) have been studied. The performance of the new technique was investigated using various experiments derived from the general mixer model and using real-time data. These studies illustrated the superiority and the generality of the new technique compared with existing methods.

An information theoretic approach to blind source separation

Blind Source Separation (BSS) deals with separating independent signals from their linear mixtures observed at different sensors. In this paper a nonlinear function based on the cost function as Kullback-Leibler divergence between joint probability density function of the source vector and its parametric model has been proposed. This cost function is equivalent to maximization of information transfer between inputs and outputs and minimization of mutual information between the components of the output vector. Derivation process becomes extremely simple due to a simple approximation. Simulations with communication signals indicate that proposed algorithm provides better accuracy. The three most popular neural algorithms (EASI, natural gradient and Bell-Sejnowski algorithm) has been compared. Effectiveness of these algorithms depends upon the nonlinear activation function. These algorithms have been evaluated with different nonlinear functions for sub-Gaussian and super-Gaussian sources.

Blind source separation using Renyi's mutual information

A blind source separation algorithm is proposed that is based on minimizing Renyi's mutual information by means of nonparametric probability density function (PDF) estimation. The two-stage process consists of spatial whitening and a series of Givens rotations and produces a cost function consisting only of marginal entropies. This formulation avoids the problems of PDF inaccuracy due to truncation of series expansion and the estimation of joint PDFs in high-dimensional spaces given the typical paucity of data. Simulations illustrate the superior efficiency, in terms of data length, of the proposed method compared to fast independent component analysis (FastICA), Comon's (1994) minimum mutual information, and Bell and Sejnowski's (1995) Infomax.

Blind separation of instantaneous mixtures of nonstationary sources

Most source separation algorithms are based on a model of stationary sources. However, it is a simple matter to take advantage of possible nonstationarities of the sources to achieve separation. This paper develops novel approaches in this direction based on the principles of maximum likelihood and minimum mutual information. These principles are exploited by efficient algorithms in both the off-line case (via a new joint diagonalization procedure) and in the on-line case (via a Newton-like procedure). Some experiments showing the good performance of our algorithms and evidencing an interesting feature of our methods are presented: their ability to achieve a kind of super-efficiency. The paper concludes with a discussion contrasting separating methods for non-Gaussian and nonstationary models and emphasizing that, as a matter of fact, "what makes the algorithms work" is-strictly speaking-not the nonstationarity itself but rather the property that each realization of the source signals has a time-varying envelope.