Terms Of Mutual Information Research Articles

A multicomponent approach to nonrigid registration of diffusion tensor images

Diffusion tensor imaging has shown promise in the early detection and diagnosis of a host of disorders and neurologic conditions. In this paper, we propose a nonrigid registration approach for diffusion tensor images using a multicomponent information-theoretic measure. Explicit orientation optimization is enabled by incorporating tensor reorientation, which is necessary for wrapping diffusion tensor images. Experimental results on diffusion tensor images indicate the feasibility of the proposed approach and a much better performance compared to the affine registration method based on mutual information in terms of registration accuracy in the presence of geometric distortion.

SU‐F‐I‐09: Improvement of Image Registration Using Total‐Variation Based Noise Reduction Algorithms for Low‐Dose CBCT

Purpose:To study the effect of total‐variation based noise reduction algorithms to improve the image registration of low‐dose CBCT for patient positioning in radiation therapy.Methods:In low‐dose CBCT, the reconstructed image is degraded by excessive quantum noise. In this study, we developed a total‐variation based noise reduction algorithm and studied the effect of the algorithm on noise reduction and image registration accuracy. To study the effect of noise reduction, we have calculated the peak signal‐to‐noise ratio (PSNR). To study the improvement of image registration, we performed image registration between volumetric CT and MV‐ CBCT images of different head‐and‐neck patients and calculated the mutual information (MI) and Pearson correlation coefficient (PCC) as a similarity metric. The PSNR, MI and PCC were calculated for both the noisy and noise‐reduced CBCT images.Results:The algorithms were shown to be effective in reducing the noise level and improving the MI and PCC for the low‐dose CBCT images tested. For the different head‐and‐neck patients, a maximum improvement of PSNR of 10 dB with respect to the noisy image was calculated. The improvement of MI and PCC was 9% and 2% respectively.Conclusion:Total‐variation based noise reduction algorithm was studied to improve the image registration between CT and low‐dose CBCT. The algorithm had shown promising results in reducing the noise from low‐dose CBCT images and improving the similarity metric in terms of MI and PCC.

Conditional and Marginal Mutual Information in Gaussian and Hyperbolic Decay Time Series

We consider the amount of available information about an arbitrary future state of a Gaussian stochastic process. We derive an infinite series for the marginal mutual information in terms of the autocorrelation function. We derive an infinite series for the newly available information for prediction, the conditional mutual information, in terms of the moving average parameters, and directly characterize predictability in terms of sensitivity to random shocks. We apply our results to long memory, or more generally, hyperbolic decay models, and give information‐theoretic characterizations of the transition from persistence to anti‐persistence, stationary long memory to nonstationarity, and a stationary regime where the mutual information is not summable.

Constrained neighborhood preserving concept factorization for data representation

Matrix factorization based techniques, such as nonnegative matrix factorization (NMF) and concept factorization (CF), have attracted a great deal of attentions in recent years, mainly due to their ability of dimension reduction and sparse data representation. Both techniques are of unsupervised nature and thus do not make use of a priori knowledge to guide the clustering process. This could lead to inferior performance in some scenarios. As a remedy to this, a semi-supervised learning method called Pairwise Constrained Concept Factorization (PCCF) was introduced to incorporate some pairwise constraints into the CF framework. Despite its improved performance, PCCF uses only a priori knowledge and neglects the proximity information of the whole data distribution; this could lead to rather poor performance (although slightly improved comparing to CF) when only limited a priori information is available. To address this issue, we propose in this paper a novel method called Constrained Neighborhood Preserving Concept Factorization (CNPCF). CNPCF utilizes both a priori knowledge and local geometric structure of the dataset to guide its clustering. Experimental studies on three real-world clustering tasks demonstrate that our method yields a better data representation and achieves much improved clustering performance in terms of accuracy and mutual information comparing to the state-of-the-arts techniques.

Information Extraction Under Privacy Constraints

A privacy-constrained information extraction problem is considered where for a pair of correlated discrete random variables (X,Y) governed by a given joint distribution, an agent observes Y and wants to convey to a potentially public user as much information about Y as possible while limiting the amount of information revealed about X. To this end, the so-called rate-privacy function is investigated to quantify the maximal amount of information (measured in terms of mutual information) that can be extracted from Y under a privacy constraint between X and the extracted information, where privacy is measured using either mutual information or maximal correlation. Properties of the rate-privacy function are analyzed and its information-theoretic and estimation-theoretic interpretations are presented for both the mutual information and maximal correlation privacy measures. It is also shown that the rate-privacy function admits a closed-form expression for a large family of joint distributions of (X,Y). Finally, the rate-privacy function under the mutual information privacy measure is considered for the case where (X,Y) has a joint probability density function by studying the problem where the extracted information is a uniform quantization of Y corrupted by additive Gaussian noise. The asymptotic behavior of the rate-privacy function is studied as the quantization resolution grows without bound and it is observed that not all of the properties of the rate-privacy function carry over from the discrete to the continuous case.

Holographic RG flows, entanglement entropy and the sum rule

We calculate the two-point function of the trace of the stress tensor in holographic renormalization group flows between pairs of conformal field theories. We show that the term proportional to the momentum squared in this correlator gives the change of the central charge between fixed points in d=2 and in d>2 it gives the holographic entanglement entropy for a planar region. This can also be seen as a holographic realization of the Adler-Zee formula for the renormalization of Newton's constant. Holographic regularization is found to provide a perfect match of the finite and divergent terms of the sum rule, and it is analogous to the regularization of the entropy in terms of mutual information. Finally, we provide a general proof of reflection positivity in terms of stability of the dual bulk action, and discuss the relation between unitarity constraints, the null energy condition and regularity in the interior of the gravity solution.

On the second-order asymptotics for entanglement-assisted communication

The entanglement-assisted classical capacity of a quantum channel is known to provide the formal quantum generalization of Shannon's classical channel capacity theorem, in the sense that it admits a single-letter characterization in terms of the quantum mutual information and does not increase in the presence of a noiseless quantum feedback channel from receiver to sender. In this work, we investigate second-order asymptotics of the entanglement-assisted classical communication task. That is, we consider how quickly the rates of entanglement-assisted codes converge to the entanglement-assisted classical capacity of a channel as a function of the number of channel uses and the error tolerance. We define a quantum generalization of the mutual information variance of a channel in the entanglement-assisted setting. For covariant channels, we show that this quantity is equal to the channel dispersion, and thus completely characterize the convergence towards the entanglement-assisted classical capacity when the number of channel uses increases. Our results also apply to entanglement-assisted quantum communication, due to the equivalence between entanglement-assisted classical and quantum communication established by the teleportation and super-dense coding protocols.

Class-driven concept factorization for image representation

Recently, concept factorization (CF), which is a variant of nonnegative matrix factorization, has attracted great attentions in image representation. In CF, each concept is modeled as a nonnegative linear combination of the data points, and each data point as a linear combination of the concepts. CF has impressive performances in data representation. However, it is an unsupervised learning method without considering the label information of the data points. In this paper, we propose a novel semi-supervised CF method, called class-driven concept factorization (CDCF), which associates the class labels of data points with their representations by introducing a class-driven constraint. This constraint forces the representations of data points to be more similar within the same class while different between classes. Thus, the discriminative abilities of the representations are enhanced in the image representation. Experimental results on several databases have shown the effectiveness of our proposed method in terms of clustering accuracy and mutual information.

Advanced density matrix renormalization group method for nuclear structure calculations

We present an efficient implementation of the Density Matrix Renormalization Group (DMRG) algorithm that includes an optimal ordering of the proton and neutron orbitals and an efficient expansion of the active space utilizing various concepts of quantum information theory. We first show how this new DMRG methodology could solve a previous $400$ KeV discrepancy in the ground state energy of $^{56}$Ni. We then report the first DMRG results in the $pf+g9/2$ shell model space for the ground $0^+$ and first $2^+$ states of $^{64}$Ge which are benchmarked with reference data obtained from Monte Carlo shell model. The corresponding correlation structure among the proton and neutron orbitals is determined in terms of the two-orbital mutual information. Based on such correlation graphs we propose several further algorithmic improvement possibilities that can be utilized in a new generation of tensor network based algorithms.

Semi-supervised concept factorization for document clustering

Nonnegative Matrix Factorization (NMF) and Concept Factorization (CF) are two popular methods for finding the low-rank approximation of nonnegative matrix. Different from NMF, CF can be applied not only to the matrix containing negative values but also to the kernel space. Based on NMF and CF, many methods, such as Graph regularized Nonnegative Matrix Factorization (GNMF) and Locally Consistent Clustering Factorization (LCCF) can significantly improve the performance of clustering. Unfortunately, these are unsupervised learning methods. In order to enhance the clustering performance with the supervisory information, a Semi-Supervised Concept Factorization (SSCF) is proposed in this paper by incorporating the pairwise constraints into CF as the reward and penalty terms, which can guarantee that the data points belonging to a cluster in the original space are still in the same cluster in the transformed space. By comparing with the state-of-the-arts algorithms (KM, NMF, CF, LCCF, GNMF, PCCF), experimental results on document clustering show that the proposed algorithm has better performance in terms of accuracy and mutual information.

Local available quantum correlations

In this work, local available quantum correlations are studied. They are defined in terms of mutual information of bipartite local measurements done over an optimal local basis complementary to the local basis which defines the respective classical correlations. For two qubits, it is always possible to choose the basis of classical correlations as the set of eigenvectors of $$\sigma _z$$?z (the third Pauli matrix) and complementary bases become the sets of eigenvectors of the observables orthogonal to $$\sigma _z$$?z. It is shown that all states with zero local available quantum correlations are separable but not necessarily strictly classical; this fact puts this kind of correlations in the middle between discord and entanglement. Since in many cases it may suffice to know whether a given state has quantum correlations, the structure of the states with zero local available quantum correlations is presented. It is also shown that there is a close connection between local available quantum correlations and the protocol of entanglement activation developed by Piani et al. (Phys Rev Lett 106:220403, 2011). If a state satisfies the sufficient condition for the entanglement swapping associated with this protocol, this state has nonzero local available quantum correlations.

Mutual information and the F-theorem

Mutual information is used as a purely geometrical regularization of entanglement entropy applicable to any QFT. A coefficient in the mutual information between concentric circular entangling surfaces gives a precise universal prescription for the monotonous quantity in the c-theorem for d = 3. This is in principle computable using any regularization for the entropy, and in particular is a definition suitable for lattice models. We rederive the proof of the c-theorem for d = 3 in terms of mutual information, and check our arguments with holographic entanglement entropy, a free scalar field, and an extensive mutual information model.

Image Forgery Detection using Multi Scale Entropy Filter and Local Phase Quantization

Image fusion is a popular application of image processing which performs merging of two or more images into one. The merged image is of improved visual quality and carries more information content. The present work introduces a new image fusion method in complex wavelet domain. The proposed fusion rule is based on a level dependent threshold, where absolute difference of a wavelet coefficient from the threshold value is taken as fusion criteria. This absolute difference represents variation in the image intensity that resembles the salient features of image. Hence, for fusion, the coefficients that are far from threshold value are being selected. The motivation behind using dual tree complex wavelet transform is due to failure of real valued wavelet transform in many aspects. Good directional selectivity, availability of phase information and approximate shift invariant nature of dual tree complex wavelet transform make it suitable for image fusion and help to produce a high quality fused image. To prove the strength of the proposed method, it has been compared with several spatial, pyramidal, wavelet and new generation wavelet based fusion methods. The experimental results show that the proposed method outperforms all the other state-of- the-art methods visually as well quantitatively as in terms of standard deviation, mutual information, edge strength, fusion factor, sharpness and average gradient.

SAR speckle reduction using Laplace mixture model and spatial mutual information in the directionlet domain

The reduction of multiplicative speckle noise which always complicates the human and automatic interpretation of objects is very significant for the practical applications of synthetic aperture radar (SAR) image. In this paper, a new maximum a posteriori (MAP) despeckling method based on directionlet transform is proposed. To convert the multiplicative noise into an additive one, the logarithmic transform is first applied to the SAR images. Then, the directionlet coefficients of the noise-free (or underlying backscatter) image and of the speckle noise are modeled as Laplace mixture distribution with zero-mean and Gaussian distribution, respectively. Within Bayesian framework, a MAP estimator is constructed using these assumed prior distributions. After obtaining the parameter estimates using expectation–maximization algorithm, the noise-free coefficients are estimated by a non-linear shrinkage function based on the average version of Bayesian estimator. To improve the denoising performance, we combine the intra-scale dependency in terms of mutual information with the MAP estimator to refine the estimated results. Finally, we compare the proposed algorithm with several other speckle filters applied on synthetic and actual SAR images. Experimental results show that the proposed method outperforms other filters in terms of signal-to-noise ratio, edge preservation and equivalent number of looks measures in most cases.

An application of mutual information in mathematical statistics education

In mathematical statistics education, we can use mutual information as a tool for evaluating the degree of dependency between two random variables. The ordinary correlation coefficient provides information only on linear dependency, not on nonlinear relationship between two random variables if any. In this paper as a measure of the degree of dependency between random variables, we suggest the use of symmetric uncertainty and <TEX>${\lambda}$</TEX> which are defined in terms of mutual information. They can be also considered as generalized correlation coefficients for both linear and non-linear dependence of random variables.

Information-Theoretic Inference of Common Ancestors

A directed acyclic graph (DAG) partially represents the conditional independence structure among observations of a system if the local Markov condition holds, that is if every variable is independent of its non-descendants given its parents. In general, there is a whole class of DAGs that represents a given set of conditional independence relations. We are interested in properties of this class that can be derived from observations of a subsystem only. To this end, we prove an information-theoretic inequality that allows for the inference of common ancestors of observed parts in any DAG representing some unknown larger system. More explicitly, we show that a large amount of dependence in terms of mutual information among the observations implies the existence of a common ancestor that distributes this information. Within the causal interpretation of DAGs, our result can be seen as a quantitative extension of Reichenbach’s principle of common cause to more than two variables. Our conclusions are valid also for non-probabilistic observations, such as binary strings, since we state the proof for an axiomatized notion of “mutual information” that includes the stochastic as well as the algorithmic version.

FPGA-Based Emulation of a Synchronous Phase-Coded Quantum Cryptography System

We present FPGA-based emulation of a synchronous phase-coded quantum cryptography system. Several of the emulated subsystems are used for implementation in a free space demonstrative QPSK scheme for quantum key distribution with continuous variables (CV-QKD) using a base and optical phase synchronization. The CV-QKD systems are commonly implemented using QPSK modulation with switched or simultaneous detection. In this paper we only make use of one base of the QPSK system in order to get a simpler modulation (BPSK) scheme, used for demonstrative purposes. The reported results from the emulation and the experiment in terms of Quantum Bit Error Rate (QBER) and mutual information for different values of the number of photons per bit are in good agreement.

Estimating the decomposition of predictive information in multivariate systems.

In the study of complex systems from observed multivariate time series, insight into the evolution of one system may be under investigation, which can be explained by the information storage of the system and the information transfer from other interacting systems. We present a framework for the model-free estimation of information storage and information transfer computed as the terms composing the predictive information about the target of a multivariate dynamical process. The approach tackles the curse of dimensionality employing a nonuniform embedding scheme that selects progressively, among the past components of the multivariate process, only those that contribute most, in terms of conditional mutual information, to the present target process. Moreover, it computes all information-theoretic quantities using a nearest-neighbor technique designed to compensate the bias due to the different dimensionality of individual entropy terms. The resulting estimators of prediction entropy, storage entropy, transfer entropy, and partial transfer entropy are tested on simulations of coupled linear stochastic and nonlinear deterministic dynamic processes, demonstrating the superiority of the proposed approach over the traditional estimators based on uniform embedding. The framework is then applied to multivariate physiologic time series, resulting in physiologically well-interpretable information decompositions of cardiovascular and cardiorespiratory interactions during head-up tilt and of joint brain-heart dynamics during sleep.

Mutual information area laws for thermal free fermions

We provide a rigorous and asymptotically exact expression of the mutual information of translationally invariant free fermionic lattice systems in a Gibbs state. In order to arrive at this result, we introduce a novel framework for computing determinants of Töplitz operators with smooth symbols, and for treating Töplitz matrices with system size dependent entries. The asymptotically exact mutual information for a partition of the 1D lattice satisfies an area law, with a prefactor which we compute explicitly. As examples, we discuss the fermionic XX model in one dimension and free fermionic models on the torus in higher dimensions in detail. Special emphasis is put on the discussion of the temperature dependence of the mutual information, scaling like the logarithm of the inverse temperature, hence confirming an expression suggested by conformal field theory. We also comment on the applicability of the formalism to treat open systems driven by quantum noise. In the appendix, we derive useful bounds to the mutual information in terms of purities. Finally, we provide a detailed error analysis for finite system sizes. This analysis is valuable in its own right for the abstract theory of Töplitz determinants.

A Trellis Shaping for Peak and Average Power Reduction of BICM-OFDM Signals

One of the major drawbacks of orthogonal frequency-division multiplexing (OFDM) systems is their high peak-to-average power ratio (PAPR) signals, and trellis shaping (TS) is a promising approach for reducing both peak power and average power of OFDM signals. In practice, OFDM should be employed with coded modulation for improving reliability, and bit-interleaved coded modulation (BICM) has been often combined with OFDM to gain robustness against severe fading channels. In this work, we propose a new BICM-OFDM system concatenated with TS for achieving good error performance. To this end, a novel soft-in-soft-out decoder for TS is developed. Optimization of the shaping convolutional code in terms of transmitter performance is also discussed. The effectiveness of our system is demonstrated through the analysis in terms of average mutual information as well as frame error rate (FER) over an additive white Gaussian noise channel. Furthermore, the FER is evaluated over a frequency-selective Rayleigh fading channel and compared with the theoretical outage probability. The simulation results reveal that our system can achieve good frequency diversity effect over such a fading channel while enjoying much lower PAPR.