Computation Of Mutual Information Research Articles

An information-theoretic characterization of partitioned property spaces

A methodology, derived by analogy to Shannon’s information-theoretic theory of communication and utilizing the concept of mutual information, has been developed to characterize partitioned property spaces. A family of non-intersecting subsets that cover the “universe” of objects represents a partitioned property space. Each subset is thus an equivalence class. A partition and it’s associated equivalence classes can be generated using any one of a number of procedures including hierarchical and non-hierarchical clustering, direct approaches using rough set methods, and cell-based partitioning, to name a few. Thus, partitioned property spaces arise in many instances and represent a very large class of problems. The approach is based on set-valued mappings from equivalence classes in one partition to those in another and provides a coarse-grained means for comparing property spaces. From these mappings it is possible to compute a number of Shannon entropies that afford calculation of mutual information, which represents that amount of information shared by two partitions of a set of objects. Taking the ratio of the mutual information with the maximum possible mutual information yields a quantity that measures the similarity of the two partitions. While the focus in this work is directed towards small sets of objects the approach can be extended to many more classes of problems that can be put into a similar form, which includes many types of cheminformatic and biological problems. A number of scenarios are presented that illustrate the concept and indicate the broader class of problems that can be handled by this method.

Statistical validation of mutual information calculations: Comparison of alternative numerical algorithms

Given two time series X and Y , their mutual information, I (X,Y) = I (Y,X) , is the average number of bits of X that can be predicted by measuring Y and vice versa. In the analysis of observational data, calculation of mutual information occurs in three contexts: identification of nonlinear correlation, determination of an optimal sampling interval, particularly when embedding data, and in the investigation of causal relationships with directed mutual information. In this contribution a minimum description length argument is used to determine the optimal number of elements to use when characterizing the distributions of X and Y . However, even when using partitions of the X and Y axis indicated by minimum description length, mutual information calculations performed with a uniform partition of the XY plane can give misleading results. This motivated the construction of an algorithm for calculating mutual information that uses an adaptive partition. This algorithm also incorporates an explicit test of the statistical independence of X and Y in a calculation that returns an assessment of the corresponding null hypothesis. The previously published Fraser-Swinney algorithm for calculating mutual information includes a sophisticated procedure for local adaptive control of the partitioning process. When the Fraser and Swinney algorithm and the algorithm constructed here are compared, they give very similar numerical results (less than 4% difference in a typical application). Detailed comparisons are possible when X and Y are correlated jointly Gaussian distributed because an analytic expression for I (X,Y) can be derived for that case. Based on these tests, three conclusions can be drawn. First, the algorithm constructed here has an advantage over the Fraser-Swinney algorithm in providing an explicit calculation of the probability of the null hypothesis that X and Y are independent. Second, the Fraser-Swinney algorithm is marginally the more accurate of the two algorithms when large data sets are used. With smaller data sets, however, the Fraser-Swinney algorithm reports structures that disappear when more data are available. Third, the algorithm constructed here requires about 0.5% of the computation time required by the Fraser-Swinney algorithm.

SU‐DD‐A4‐02: Fast Multimodal Image Registration Using Mutual Information and Variable Step‐Size Grid Search Algorithm

Purpose: A robust optimization algorithm, referred to as variable step‐size grid search (VSGS), is presented for 2D/2D multimodal image registration, and evaluated using phantom and clinical imaging. Method and Materials: The VSGS optimization algorithm is based on recursive iterations of a geometrically scaled step‐size, and is more likely to converge to the global optimum and be less insensitive to the local optima compared to traditional gradient techniques (i.e. Powell, quasi‐Newton). As an optimization technique, it can be used with any objective function. Here mutual information was used as the similarity measure, and a multiresolution strategy was adopted to accelerate computation without adversely affecting robustness. VSGS was applied to the problem of automatic patient positioning in radiotherapy. The multimodal imaging consists of digitally reconstructed radiographs (DRRs), portal images acquired with electronic portal imaging devices (EPID), and the Varian linac‐based on‐board kilovoltage imager. EPID images were collected using the Siemens BEAMVIEW video EPID and the Varian AS500 flat‐panel imager. Results: Validation was carried out using phantom and clinical imaging data. Measurements indicated that VSGS has an accuracy of 0.5 mm (or 1 pixel) and <0.5°, and required about 8 seconds per registration on a 2 GHz PC. VSGS has an accuracy equivalent to or better than the traditional optimization techniques depending on histogram or Parzen‐windowing based mutual information calculations, but is more robust and faster computationally. This allows VSGS to be used for online real‐time image guided patient positioning without requiring frequent user intervention to recognize shifts associated with registrations trapped in local extrema. Conclusion: VSGS can be used for online real‐time image guided patient positioning to achieve more accurate patient positioning than using visual verification, and is sufficiently robust compared to traditional optimization techniques to make algorithm‐based registrations sufficiently robust for routine clinical use.

Hardware Acceleration of Mutual Information-Based 3D Image Registration

Real-time image registration is potentially an enabling technology for the effective and efficient use of many image-guided diagnostic and treatment procedures relying on multimodality image fusion or serial image comparison. Mutual information is currently the best-known image similarity measure for intensity-based multimodality image registration. The calculation of mutual information is memory intensive and does not benefit from cache-based memory architectures in standard software implementations, i.e., the calculation incurs a large number of cache misses. Previous attempts to perform image registration in real time focused on parallel supercomputer implementations, which achieved significant speedups using large, expensive supercomputers that are impractical for clinical deployment. We present a hardware architecture that can be used to accelerate a number of linear and elastic image registration algorithms that use mutual information as an image similarity measure. A proof-of-concept implementation of the architecture achieved speedups of 30 for linear registration and 100 for elastic registration against a 3.2 GHz Pentium III Xeon workstation. Further speedup can be achieved by using several modules in parallel.

Does independent component analysis play a role in unmixing hyperspectral data?

Independent component analysis (ICA) has recently been proposed as a tool to unmix hyperspectral data. ICA is founded on two assumptions: 1) the observed spectrum vector is a linear mixture of the constituent spectra (endmember spectra) weighted by the correspondent abundance fractions (sources); 2)sources are statistically independent. Independent factor analysis (IFA) extends ICA to linear mixtures of independent sources immersed in noise. Concerning hyperspectral data, the first assumption is valid whenever the multiple scattering among the distinct constituent substances (endmembers) is negligible, and the surface is partitioned according to the fractional abundances. The second assumption, however, is violated, since the sum of abundance fractions associated to each pixel is constant due to physical constraints in the data acquisition process. Thus, sources cannot be statistically independent, this compromising the performance of ICA/IFA algorithms in hyperspectral unmixing. This paper studies the impact of hyperspectral source statistical dependence on ICA and IFA performances. We conclude that the accuracy of these methods tends to improve with the increase of the signature variability, of the number of endmembers, and of the signal-to-noise ratio. In any case, there are always endmembers incorrectly unmixed. We arrive to this conclusion by minimizing the mutual information of simulated and real hyperspectral mixtures. The computation of mutual information is based on fitting mixtures of Gaussians to the observed data. A method to sort ICA and IFA estimates in terms of the likelihood of being correctly unmixed is proposed.

Study of acoustic correlates associate with emotional speech

This study investigates the acoustic characteristics of four different emotions expressed in speech. The aim is to obtain detailed acoustic knowledge on how a speech signal is modulated by changes from neutral to a certain emotional state. Such knowledge is necessary for automatic emotion recognition and classification and emotional speech synthesis. Speech data obtained from two semi-professional actresses are analyzed and compared. Each subject produces 211 sentences with four different emotions; neutral, sad, angry, happy. We analyze changes in temporal and acoustic parameters such as magnitude and variability of segmental duration, fundamental frequency and the first three formant frequencies as a function of emotion. Acoustic differences among the emotions are also explored with mutual information computation, multidimensional scaling and acoustic likelihood comparison with normal speech. Results indicate that speech associated with anger and happiness is characterized by longer duration, shorter interword silence, higher pitch and rms energy with wider ranges. Sadness is distinguished from other emotions by lower rms energy and longer interword silence. Interestingly, the difference in formant pattern between [happiness/anger] and [neutral/sadness] are better reflected in back vowels such as /a/(/father/) than in front vowels. Detailed results on intra- and interspeaker variability will be reported.

Fast and accurate image registration using Tsallis entropy and simultaneous perturbation stochastic approximation

The Tsallis measure of mutual information is combined with the simultaneous perturbation stochastic approximation algorithm to register images. It is shown that Tsallis entropy can improve registration accuracy and speed of convergence, compared with Shannon entropy, in the calculation of mutual information. Simulation results show that the new algorithm achieves up to seven times faster convergence and four times more precise registration than using a classic form of entropy.

Extending the mutual information measure to rank inferred literature relationships.

BackgroundWithin the peer-reviewed literature, associations between two things are not always recognized until commonalities between them become apparent. These commonalities can provide justification for the inference of a new relationship where none was previously known, and are the basis of most observation-based hypothesis formation. It has been shown that the crux of the problem is not finding inferable associations, which are extraordinarily abundant given the scale-free networks that arise from literature-based associations, but determining which ones are informative. The Mutual Information Measure (MIM) is a well-established method to measure how informative an association is, but is limited to direct (i.e. observable) associations.ResultsHerein, we attempt to extend the calculation of mutual information to indirect (i.e. inferable) associations by using the MIM of shared associations. Objects of general research interest (e.g. genes, diseases, phenotypes, drugs, ontology categories) found within MEDLINE are used to create a network of associations for evaluation.ConclusionsMutual information calculations can be effectively extended into implied relationships and a significance cutoff estimated from analysis of random word networks. Of the models tested, the shared minimum MIM (MMIM) model is found to correlate best with the observed strength and frequency of known associations. Using three test cases, the MMIM method tends to rank more specific relationships higher than counting the number of shared relationships within a network.

Reliable prediction of T-cell epitopes using neural networks with novel sequence representations.

In this paper we describe an improved neural network method to predict T-cell class I epitopes. A novel input representation has been developed consisting of a combination of sparse encoding, Blosum encoding, and input derived from hidden Markov models. We demonstrate that the combination of several neural networks derived using different sequence-encoding schemes has a performance superior to neural networks derived using a single sequence-encoding scheme. The new method is shown to have a performance that is substantially higher than that of other methods. By use of mutual information calculations we show that peptides that bind to the HLA A*0204 complex display signal of higher order sequence correlations. Neural networks are ideally suited to integrate such higher order correlations when predicting the binding affinity. It is this feature combined with the use of several neural networks derived from different and novel sequence-encoding schemes and the ability of the neural network to be trained on data consisting of continuous binding affinities that gives the new method an improved performance. The difference in predictive performance between the neural network methods and that of the matrix-driven methods is found to be most significant for peptides that bind strongly to the HLA molecule, confirming that the signal of higher order sequence correlation is most strongly present in high-binding peptides. Finally, we use the method to predict T-cell epitopes for the genome of hepatitis C virus and discuss possible applications of the prediction method to guide the process of rational vaccine design.

Nonlinear analysis of a rectangular natural circulation loop

The aim of this paper is to present a novel approach to the analysis of the dynamical evolution of a rectangular natural circulation loop heated from below. During an experimental phase nonlinear temperature oscillations were detected at different location and for various unstable operating conditions of the loop. The experimental time series were intially filtered using wavelet packet approach. Afterwards, Singular Value Decomposition (SVD) was applied, so to reduce the filtered signals to their dominant features, and the calculation of the Average Mutual Information (AMI) function was performed. Both these approaches were adopted in order both to take into account the intrinsic nonlinear behavior of detected temperatures and to obtain a simplified phase space representation of the dynamics of the system. Such a representation was used to determine the System Dimension, whose knowledge represents the first step in the analysis of any dynamical system. Finally, phase space representations of various operating conditions show the presence of dynamical behaviors characterized by different values of the system dimension, pointing out the possible existence of chaos in the system in study.

An Information Theory Framework for the Analysis of Scene Complexity

In this paper we present a new framework for the analysis of scene visibility and radiosity complexity. We introduce a number of complexity measures from information theory quantifying how difficult it is to compute with accuracy the visibility and radiosity in a scene. We define the continuous mutual information as a complexity measure of a scene, independent of whatever discretisation, and discrete mutual information as the complexity of a discretised scene. Mutual information can be understood as the degree of correlation or dependence between all the points or patches of a scene. Thus, low complexity corresponds to low correlation and vice versa. Experiments illustrating that the best mesh of a given scene among a number of alternatives corresponds to the one with the highest discrete mutual information, indicate the feasibility of the approach. Unlike continuous mutual information, which is very cheap to compute, the computation of discrete mutual information can however be quite demanding. We will develop cheap complexity measure estimates and derive practical algorithms from this framework in future work.

Enlarged similarity of nucleic acid sequences.

The concept of nucleic acid sequence base alternations is presented. The number of base alterations for the sequences of different length is established. The definition of "enlarged similarity" of nucleic acids sequences on the basis of sequence base alterations is introduced. Mutual information between sequences is used as a quantitative measure of enlarged similarity for two compared sequences. The method of mutual information calculation is developed considering the correlation of bases in compared sequences. The definitions of correlated similarity and evolution similarity between compared sequences are given. Results of the use of enlarged similarity approach for DNA sequences analysis are discussed.

Determination of eukaryotic protein coding regions using neural networks and information theory

Our previous work applied neural network techniques to the problem of discriminating open reading frame (ORF) sequences taken from introns versus exons. The method counted the codon frequencies in an ORF of a specified length, and then used this codon frequency representation of DNA fragments to train a neural net (essentially a Perceptron with a sigmoidal, or “soft step function”, output) to perform this discrimination. After training, the network was then applied to a disjoint “predict” set of data to assess accuracy. The resulting accuracy in our previous work was 98.4%, exceeding accuracies reported in the literature at that time for other algorithms. Here, we report even higher accuracies stemming from calculations of mutual information (a correlation measure) of spatiallly separated codons in exons, and in introns. Significant mutual information exists in exons, but not in introns, between adjacent codons. This suggests that dicodon frequencies of adjacent codons are important for intron/exon discrimination. We report that accuracies obtained using a neural net trained on the frequency of dicodons is significantly higher at smaller fragment lengths than even our original results using codon frequencies, which were already higher than simple statistical methods that also used codon frequencies. We also report accuracies obtained from including codon and dicodon statistics in all six reading frames, i.e. the three frames on the original and complement strand. Inclusion of six-frame statistics increases the accuracy still further. We also compare these neural net results to a Bayesian statistical prediction method that assumes independent codon frequencies in each position. The performance of the Bayesian scheme is poorer than any of the neural based schemes, however many methods reported in the literature either explicitly, or implicitly, use this method. Specifically, Bayesian prediction schemes based on codon frequencies achieve 90.9% accuracy on 90 codon ORFs, while our best neural net scheme reaches 99.4% accuracy on 60 codon ORFs. “Accuracy” is defined as the average of the exon and intron sensitivities. Achievement of sufficiently high accuracies on short fragment lengths can be useful in providing a computational means of finding coding regions in unannotated DNA sequences such as those arising from the mega-base sequencing efforts of the Human Genome Project. We caution that the high accuracies reported here do not represent a complete solution to the problem of identifying exons in “raw” base sequences. The accuracies are considerably lower from exons of small length, although still higher than accuracies reported in the literature for other methods. Short exon lengths are not uncommon. A complete solution to the problem may need a combination of methods including accurate, computational methods of identifying splice sites.

On the Calculation of Mutual Information

Abstract : Calculating the amount of information about a random function contained in another random function has important uses in communication theory. An expression for the mutual information for continuous time random processes has been given by Gelfand and Yaglom, Chiang, and Perez by generalizing Shannon's result in a natural way. Under a condition of absolute continuity of measures the continuous time expression has the same form as Shannon's result. For two Gaussian processes Gelfand and Yaglom express the mutual information in terms of a mean square estimation error. We generalize this result to diffusion processes and express the solution in a different form which is more naturally related to a corresponding filtering problem. We also use these results to calculate some information rates.

The innovations approach to detection and estimation theory

Given a stochastic process, its innovations process will be defined as a white Gaussian noise process obtained from the original process by a causal and causally invertible transformation. The significance of such a representation, when it exists, is that statistical inference problems based on observation of the original process can be replaced by simpler problems based on white noise observations. Seven applications to linear and nonlinear least-squares estimation. Gaussian and non-Gaussian detection problems, solution of Fredholm integral equations, and the calculation of mutual information, will be described. The major new results are summarized in seven theorems. Some powerful mathematical tools will be introduced, but emphasis will be placed on the considerable physical significance of the results.

The Capacity of Linear Channels with Additive Gaussian Noise

The standard method of computing the mutual information between two stochastic processes with finite energy replaces the processes with their Fourier coefficients. This procedure is mathematically justified here for random signals w,(ω) square-integrable in the product space t × ω where t ∊ [O, T] and w is an element of a probability space. A natural notion of the sigma field generated by w, (ω) is presented and it is shown to coincide with the sigma field generated by the random Fourier coefficients of w,(ω) in any complete orthonormal system in L <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</inf> [O, T]. This justifies the use of Fourier coefficients in mutual information computations. Capacity is calculated for finite and infinite-dimensional channels, where the output signal consists of a filter (general Hilbert-Schmidt operator) operating on the input signal with additive Gaussian noise. The finite-dimensional optimal signal is obtained. In the infinite-dimensional case capacity can be approached arbitrarily closely with finite-dimensional inputs. The question of the existence of an infinite-dimensional signal which achieves capacity is considered. There are channels for which no signal achieves capacity. Some results are obtained when the noise coordinates are independent in the eigensystem of the filter.