Mutual Information Principle Research Articles

Application of the mutual information principle to spectral density estimation

The power spectrum of a stationary Gaussian random process is estimated when partial knowledge of the autocorrelation function is available {\em a priori}. Particular attention is paid to the case when the {\em a priori} knowledge is not precise, i.e., when there are errors in the measurements, perhaps due to the presence of noise. In the special case when the {\em a priori} knowledge consists of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</tex> points of the autocorrelation function, Burg's method of picking the spectrum which maximizes the entropy of the Gaussian process has been recently extended by Newman to account for a weighted average error in the estimates of the correlation function points. A new method is suggested here that uses the mutual information principle (MIP) of Tzannes and Noonan. The first <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</tex> points of the correlation function (obtained with errors) are used to derive an approximate spectrum by Burg's or any other method. This spectrum, as well as the error constraints involved, is then used to arrive at the underlying spectrum in the framework of the MIP approach.

On reconstituting a process from its quantized form in the presence of noise

The process of quantization is, of course, irreversible, and the probabilistic model of the input to the quantizer cannot be generally derived by knowledge of the output, even in the absence of noise. In this paper, solutions are found to this problem in the presence of noise (not necessarily additive). These solutions are derived on the assumption that the following are known; 1. (a) the probabilistic model of the output discrete random variable Y, 2. (b) the partitioning of the input X, which leads to the known values of Y, 3. (c) some error function between X and Y, which expresses the average effect of the noise. With this assumption, the problem is formulated in the context of the mutual information principle of Tzannes and Noonan. The solution leads to a partial knowledge of the input model, which is then completely derived under various constraints on the nature of the model, using a discrete version of Shannon's sampling theorem and the discrete Fourier transform properties. The method is illustrated with examples using additive noise.

The mutual information principle and applications

A general theory of prior probability models is presented, valid for both discrete and continuous random variables, even when the prior information about them has been obtained with errors. An example is included as an illustration.