Relative Information Measure Research Articles

Dynamic Quantization and Power Allocation for Multisensor Estimation of Hidden Markov Models

This paper investigates an optimal quantizer design problem for multisensor estimation of a hidden Markov model (HMMs) whose description depends on unknown parameters. The sensor measurements are simply binary quantized and transmitted to a remote fusion center over noisy flat fading wireless channels under an average sum transmit power constraint. The objective is to determine a set of optimal quantization thresholds and sensor transmit powers, called an optimal policy, which minimizes the long run average of a weighted combination of the expected state estimation error and sum transmit power. We analyze the problem by formulating an adaptive Markov decision process (MDP) problem. In this framework, adaptive optimal control policies are obtained using a nonstationary value iteration (NVI) scheme and are termed as NVI-adaptive policies. These NVI-adaptive policies are adapted to the HMM parameter estimates obtained via a strongly consistent maximum likelihood estimator. In particular, HMM parameter estimation is performed by a recursive expectation-maximization (EM) algorithm which computes estimates of the HMM parameters by maximizing a relative entropy information measure using the received quantized observations and the trajectory of the MDP. Under some regularity assumptions on the observation probability distributions and a geometric ergodicity condition on an extended Markov chain, the maximum-likelihood estimator is shown to be strongly consistent. It is shown that the NVI-adaptive policy based on this sequence of strongly consistent HMM parameter estimates is (asymptotically, under appropriate assumptions) average-optimal. Essentially, it minimizes the long run average cost of the weighted combination of the expected state estimation error and sum transmit power across the sensors for the HMM with true parameters in a time-asymptotic sense. The advantage of this scheme is that the policies are obtained recursively without the need to solve the Bellman equation at each time step, which can be computationally prohibitive. As is usual with value iteration schemes, practical implementation of the NVI-adaptive policy requires discretization of the state and action space, which results in some loss of optimality. Nevertheless, numerical results illustrate the asymptotic convergence properties of the parameter estimates and the asymptotically close to optimal performance of the adaptive MDP algorithm compared to the performance of an MDP based dynamic quantization and power allocation algorithm designed with perfect knowledge of the true parameters.

Information, data dimension and factor structure

This paper employs concepts from information theory for choosing the dimension of a data set. We propose a relative information measure connected to Kullback–Leibler numbers. By ordering the series of the data set according to the measure, we are able to obtain a subset of a data set that is most informative. The method can be used as a first step in the construction of a dynamic factor model or a leading index, as illustrated with a Monte Carlo study and with the US macroeconomic data set of Stock and Watson [20].

Mutual Information Analysis with Similarity Measure

Discussion and analysis about relative mutual information has been carried out through fuzzy entropy and similarity measure. Fuzzy relative mutual information measure (FRIM) plays an important part as a measure of information shared between two fuzzy pattern vectors. This FRIM is analyzed and explained through similarity measure between two fuzzy sets. Furthermore, comparison between two measures is also carried out.

Assessing terrorist risk and FDI using relative information measures

Recent research has explored the determinants of terrorism and the impact that terrorist activity has on a country. However, to date, no research has examined what specific aspect of terrorist risk most significantly impacts a country's ability to operate in the global market. Using the five sub-indices of the World Market Research Center's Global Terrorism Index, this study explores how the different dimensions of terrorist risk impact the inflow of Foreign Direct Investment (FDI) using a Shannon information measure of relationship. The results of this research suggest that a country's proven ability to prevent terrorist activity has the most significant impact on investment decisions in regard to FDI.

Elementary differences between the degrees of unsolvability and degrees of compressibility

Given two infinite binary sequences A , B we say that B can compress at least as well as A if the prefix-free Kolmogorov complexity relative to B of any binary string is at most as much as the prefix-free Kolmogorov complexity relative to A , modulo a constant. This relation, introduced in Nies (2005) [14] and denoted by A ≤ L K B , is a measure of relative compressing power of oracles, in the same way that Turing reducibility is a measure of relative information. The equivalence classes induced by ≤ L K are called L K degrees (or degrees of compressibility) and there is a least degree containing the oracles which can only compress as much as a computable oracle, also called the ‘low for K ’ sets. A well-known result from Nies (2005) [14] states that these coincide with the K-trivial sets, which are the ones whose initial segments have minimal prefix-free Kolmogorov complexity. We show that with respect to ≤ L K , given any non-trivial Δ 2 0 sets X , Y there is a computably enumerable set A which is not K-trivial and it is below X , Y . This shows that the local structures of Σ 1 0 and Δ 2 0 Turing degrees are not elementarily equivalent to the corresponding local structures in the LK degrees. It also shows that there is no pair of sets computable from the halting problem which forms a minimal pair in the LK degrees; this is sharp in terms of the jump, as it is known that there are sets computable from 0 ″ which form a minimal pair in the LK degrees. We also show that the structure of LK degrees below the LK degree of the halting problem is not elementarily equivalent to the Δ 2 0 or Σ 1 0 structures of LK degrees. The proofs introduce a new technique of permitting below a Δ 2 0 set that is not K -trivial, which is likely to have wider applications.

Relativistic effects on information measures for hydrogen-like atoms

Position and momentum information measures are evaluated for the ground state of the relativistic hydrogen-like atoms. Consequences of the fact that the radial momentum operator is not self-adjoint are explicitly studied, exhibiting fundamental shortcomings of the conventional uncertainty measures in terms of the radial position and momentum variances. The Shannon and Rényi entropies, the Fisher information measure, as well as several related information measures, are considered as viable alternatives. Detailed results on the onset of relativistic effects for low nuclear charges, and on the extreme relativistic limit, are presented. The relativistic position density decays exponentially at large r , but is singular at the origin. Correspondingly, the momentum density decays as an inverse power of p . Both features yield divergent Rényi entropies away from a finite vicinity of the Shannon entropy. While the position space information measures can be evaluated analytically for both the nonrelativistic and the relativistic hydrogen atom, this is not the case for the relativistic momentum space. Some of the results allow interesting insight into the significance of recently evaluated Dirac–Fock vs. Hartree–Fock complexity measures for many-electron neutral atoms.

Quantifying the Fraction of Missing Information for Hypothesis Testing in Statistical and Genetic Studies

Many practical studies rely on hypothesis testing procedures applied to data sets with missing information. An important part of the analysis is to determine the impact of the missing data on the performance of the test, and this can be done by properly quantifying the relative (to complete data) amount of available information. The problem is directly motivated by applications to studies, such as linkage analyses and haplotype-based association projects, designed to identify genetic contributions to complex diseases. In the genetic studies the relative information measures are needed for the experimental design, technology comparison, interpretation of the data, and for understanding the behavior of some of the inference tools. The central difficulties in constructing such information measures arise from the multiple, and sometimes conflicting, aims in practice. For large samples, we show that a satisfactory, likelihood-based general solution exists by using appropriate forms of the relative Kullback--Leibler information, and that the proposed measures are computationally inexpensive given the maximized likelihoods with the observed data. Two measures are introduced, under the null and alternative hypothesis respectively. We exemplify the measures on data coming from mapping studies on the inflammatory bowel disease and diabetes. For small-sample problems, which appear rather frequently in practice and sometimes in disguised forms (e.g., measuring individual contributions to a large study), the robust Bayesian approach holds great promise, though the choice of a general-purpose "default prior" is a very challenging problem.

Novel fuzzy information proximity measures

As a measure of information shared between two fuzzy pattern vectors, the fuzzy information proximity measure (FIPM) plays an important part in fuzzy pattern recognition, fuzzy clustering analysis and fuzzy approximate reasoning. In this paper, two novel FIPMs are set up. Firstly, an axiom theory about the FIPM is given, and different expressions of the FIPM are discussed. A new FIPM is then proposed based on the axiom theory of the FIPM and the concept of fuzzy subsethood function. Two concepts based on the idea of Shannon information entropy, fuzzy joint entropy (FJE) and fuzzy conditional entropy (FCE), are proposed and the basic properties of FJE and FCE are given and proved. Finally, classical similarity measures such as dissimilarity measure (DM) and similarity measure (SM) are studied, and two new measures, fuzzy absolute information measure (FAIM) and fuzzy relative information measure (FRIM), are set up, which can be used as measures of the proximity between fuzzy sets A and B.

Effects of Noise Correlations on Information Encoding and Decoding

Response variability is often correlated across populations of neurons, and these noise correlations may play a role in information coding. In previous studies, this possibility has been examined from the encoding and decoding perspectives. Here we used d prime and related information measures to examine how studies of noise correlations from these two perspectives are related. We found that for a pair of neurons, the effect of noise correlations on information decoding can be zero when the effect of noise correlations on the information encoded obtains its largest positive or negative values. Furthermore, there can be no effect of noise correlations on the information encoded when it has an effect on information decoding. We also measured the effect of noise correlations on information encoding and decoding in simultaneously recorded neurons in the supplementary motor area to see how well d prime accounted for the information actually present in the neural responses and to see how noise correlations affected encoding and decoding in real data. These analyses showed that d prime provides an accurate measure of information encoding and decoding in our population of neurons. We also found that the effect of noise correlations on information encoding was somewhat larger than the effect of noise correlations on information decoding, but both were relatively small. Finally, as predicted theoretically, the effects of correlations were slightly greater for larger ensembles (3-8 neurons) than for pairs of neurons.

A comparison between microsatellite and single-nucleotide polymorphism markers with respect to two measures of information content

Using the Genetic Analysis Workshop 14 (GAW14) simulated dataset, we compare microsatellite and single-nucleotide polymorphism (SNP) markers in terms of two measures of information content, the traditional entropy-based information content measure, and a new "relative information" measure. Both attempt to measure the amount of information contained in the markers about the identity-by-descent (IBD) sharing among relatives. The performance of the two information measures are compared based on their variability and ability to predict change in the LOD score (ΔLOD) as map density increases for SNP markers. Although in a linked region, LOD scores are correlated with measures of information, we observe that none of the measures predict the LOD score itself very well. In an unlinked region, the LOD score is not related to either measures of information. The information content of microsatellite markers with 7.5-cM spacing is slightly higher than that of SNP markers with 3-cM spacing. At these map densities, microsatellites are found to be uniformly more informative than SNPs irrespective of their level of heterozygosity. For SNPs, we found that as the level of heterozygosity increases, the information content increases. As reported in all other previous studies, we also found that high-density SNPs have higher information content compared to low-density microsatellites. Performance of both the two information measures considered here are similar, but the relative information measure predicts ΔLOD as marker density increases better than the traditional entropy-based information measure.

Identifying spatial relationships in neural processing using a multiple classification approach

The application of statistical classification methods to in vivo functional neuroimaging data makes it possible to explore spatial patterns in task-related changes in neural processing. Cluster analysis is one group of descriptive statistical procedures that can assist in identifying classes of brain regions that exhibit similar task-related functionality. In practice, a limitation of cluster analysis is that the performances of clustering algorithms rely on unknown characteristics of the data, making it difficult to determine which procedure best suits a particular analysis. We present a multiple classification approach that incorporates numerous algorithms, evaluates the associated classifications, and either selects a plausible partition relative to the others considered or pools the results from the numerous methods. The multiple classification approach utilizes a new performance criterion, called the relative information (RI) measure, to evaluate the quality of the candidate partitions and as the basis for producing a composite classification image. Employing multiple classifications, rather than a single algorithm, our methodology increases the chance of detecting the functional relationships within the data and, therefore, produces more reliable results. We apply our methodology to a PET study to explore spatial relationships in measured brain function associated with increasing blood alcohol concentration levels, and we perform a simulation study to evaluate the performance of RI.

Two Information Measures for Inconsistent Sets

I present two measures of information for both consistent and inconsistent sets of sentences in a finite language of propositional logic. The measures of information are based on measures of inconsistency developed in Knight (2002). Relative information measures are then provided corresponding to the two information measures.

On `useful' relative information and J-diver-gence measures

In the present paper some new generalized measures of useful relative information have been defined and their particular cases have been studied. From these measures new useful information measures have also been derived. We have obtained useful measures of inaccuracy, dependence and J-divergence corresponding to each measure of useful relative information. In the end, an equality satisfied by useful J-divergence has been proved.

Relative complexity changes in time series using information measures

Estimation of complexity is of great interest in nonlinear signal and system analysis. Several complexity measures have been proposed: Lyapunov exponents, Lempel and Ziv, approximate entropy. In the present study, complexity measures derived from Shannon entropy, Harvda–Charvat–Daróvczy–Tsallis ( q-entropies) and their corresponding relative information measures are presented and evaluated in the context of nonlinear systems presenting abrupt complexity changes. The performance of the proposed measures in the presence of controlled complexity is evaluated through numerical experiments using nonlinear models. An example with heart rate variability signals is presented. The results obtained show that the entropic and the relative complexity measures approach allow to discern complexity changes in a similar qualitative way compared against classical techniques but with much less computational cost and less amount of data. In the presence of noise, the relative complexity measures behave as robust tools for relative complexity changes detection. Time-scale complexity analyses are presented using the continuous multiresolution entropies. The assessment of time-scale complexity changes is also discussed.

On the relationship between the Fisher-Frieden-Soffer arrow of time, and the behaviour of the Boltzmann and Kullback entropies

We provide a new derivation of the H-theorem satisfied by Fisher's information measure ( I) for the diffusion equation and show that the ensuing monotonically decreasing behaviour of I yields, in turn, an H-theorem involving the second time derivative of the standard Boltzmann-Gibbs-Shannon entropy S. An upper bound for S( t) is obtained from this latter relation. Additionally, it is demonstrated that, under certain conditions, the behaviour of I may be related to that of the relative information measure of Kullback. This latter result is generalized to the case of the Fokker-Planck equation.

Combining ranked mean value forecasts

A methodology is developed for combining mean value forecasts using not only all the important statistics related to the past performance and the dependence of the individual forecasts, but also a rank ordering of the individual forecasts representing the belief of a decision maker about the future performance of the forecasts. The maximum likelihood combination of the forecasts turns out to be weighted linear combination of the individual forecasts, where the weights are a function of the rank order of the forecasts, correlation coefficients between the forecasts, and relative entropy information measures between the individual forecasts and the actual values. These weights are assessed once in the most general case and once in a special case where the forecasts are normally distributed. The sensitivity of the weights is also investigated. A sample application of this method for predicting U.S. hog prices is also presented.

Related information measures for the associations of Earth-science variables: Guocheng Pan, Mathematical Geology, 27(5), 1995, pp 609–632

Related information measures for the associations of Earth-science variables: Guocheng Pan, Mathematical Geology, 27(5), 1995, pp 609–632

Related information measures for the associations of earth-science variables

A basic task in earth-science data integration is to quantify variable associations. Although manv measures have been used to determine the associations between quantitative variables, the ability to quantify qualitative attributes (e.g., categorical) is limited. Moreover, most traditional association measures are restricted to linear correlations or similarities, for example, correlation coefficient. The measures proposed in this report are designed on the basis of Shannon's entropy concepts, including directional related information, ordinary related information, and partial related information. The directional related information quantifies the association of one variable in terms of another. The ordinary related information determines the mutual association of two variables. The partial related information characterizes the association of an individual stale of one variable in terms of another variable. The properties of these measures are discussed and their sample estimates are derived from both maximum likelihood and Bayesian methods. The relations between these measures are illustrated by using synthetic examples. Two applications of these measures also are developed, including the selection of variables and evaluation of mineral resources. Finally, a case study is given to demonstrate the use of the measures in mineral resources evaluation.

MEAN VALUE CHARACTERIZATION OF 'USEFUL' INFORMATION MEASURES

In the present communication the generalized mean value characterization of 'useful' information and relativeinformation measures has been studied. Some comparison theorems related to these measures have also been proved.

Information of deterministic functions and quantum information of non probabilistic matrices A unified approach and new results

This paper provides new results on the entropy of functions on the one hand, and exhibits a unified approach to entropy of functions and quantum entropy of matrices with or without probability. The entropy of continuously differentiable functions is extended to stair-wise functions, a measure of relative information between two functions is obtained, which is fully consistent with Kullback cross-entropy, Renyi cross-entropy and Fisher information. The theory is then applied to stochastic processes to yield some concepts of random geometrical entropies defined on path space, which are related to fractal dimension in the special case when the process is a fractional Brownian. Then it shows how one can obtain Shannon entropy of random variables by combining the maximum entropy principle with Hartley entropy. Lastly quantum entropy of non probabilistic matrices (extension of Von Neumann quantum mechanical entropy) is derived as a consequence of Shannon entropy of random variables.