Definition Of Mutual Information Research Articles

An integrated information theory index using multichannel EEG for evaluating various states of consciousness under anesthesia

Background:The integrated information theory (IIT) of consciousness introduces a measure Φ to quantify consciousness in a physical system. Directly related to this, general anesthesia aims to induce reversible and safe loss of consciousness (LOC). We sought to propose an electroencephalogram (EEG)-based IIT index ΦEEG to evaluate various states of consciousness under general anesthesia. Methods:Based on the definition of mutual information, we estimated the ΦEEG by maximizing the integrated information under various time lags. We used the binning method to cut the nonGaussian EEG data for estimating mutual information. We tested two EEG databases collected from propofol- (n=20) and sevoflurane-induced (n=15) anesthesia, and especially, we compared the ΦEEG of drowsy (n=7) and responsive participants (n=13) under propofol anesthesia. We compared the effectiveness of ΦEEG with the estimated bispectral index (eBIS). Results:In all EEG frequency bands, we observed a negative correlation between ΦEEG and end-tidal sevoflurane concentration under sevoflurane-induced anesthesia (p<0.001,BF10>6000). Under propofol-induced anesthesia, drowsy participants in moderate sedation (6.96±0.26(mean±SD)) showed decreased alpha-band ΦEEG compared with baseline (7.40±0.53,p=0.016,BF10=3.58), no significant difference was observed for responsive participants. Oppositely, the responsive participants in moderate sedation (−5.32±0.38) showed decreased eBIS compared with baseline (−4.94±0.40,p=0.03,BF10=2.41). Conclusions:These findings may enable monitors of the anesthetic state that can distinguish consciousness and unconsciousness rather than the changes of anesthetic concentrations. The alpha-band ΦEEG is promising for deriving the gold standard for depth of anesthesia monitoring.

Información Mutua: Una forma de cuantificar las correlaciones

Within the framework of Information Theory, the existence of correlations between two random variables means that we can obtain information about one of them, just by measuring or observing the other random variable. In certain cases, this kind of relationship allows obtaining information about a variable even when the other is separated by a very large distance, that is, the process of obtaining information is non-local, an example (if not the only) is the quantum entanglement. These features of correlations make it interesting and important to study, classify and quantify them. The correlations are classified into classical correlations and quantum correlations, in addition they are quantified through the mutual information. Here we will present a natural way to define classical mutual information and then we will generalize it to the quantum case. Furthermore, every term in the definitions of mutual information will be interpreted using the concepts of classical and quantum entropy.

Inferring functional connectivity through graphical directed information

Objective. Accurate inference of functional connectivity is critical for understanding brain function. Previous methods have limited ability distinguishing between direct and indirect connections because of inadequate scaling with dimensionality. This poor scaling performance reduces the number of nodes that can be included in conditioning. Our goal was to provide a technique that scales better and thereby enables minimization of indirect connections. Approach. Our major contribution is a powerful model-free framework, graphical directed information (GDI), that enables pairwise directed functional connections to be conditioned on the activity of substantially more nodes in a network, producing a more accurate graph of functional connectivity that reduces indirect connections. The key technology enabling this advancement is a recent advance in the estimation of mutual information (MI), which relies on multilayer perceptrons and exploiting an alternative representation of the Kullback–Leibler divergence definition of MI. Our second major contribution is the application of this technique to both discretely valued and continuously valued time series. Main results. GDI correctly inferred the circuitry of arbitrary Gaussian, nonlinear, and conductance-based networks. Furthermore, GDI inferred many of the connections of a model of a central pattern generator circuit in Aplysia, while also reducing many indirect connections. Significance. GDI is a general and model-free technique that can be used on a variety of scales and data types to provide accurate direct connectivity graphs and addresses the critical issue of indirect connections in neural data analysis.

Information-Theoretic Interactive Sensing and Inference for Autonomous Systems

This paper addresses an autonomous exploration problem in which a mobile sensor, placed in a previously unseen search area, utilizes an information-theoretic navigation cost function to dynamically select the next sensing action, i.e., location from which to take a measurement, to efficiently detect and classify objects of interest within the area. The information-theoretic cost function proposed in this paper consist of two <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">information gain</i> terms, one for detection and localization of objects and the other for sequential classification of the detected objects. We present a novel closed-form representation for the cost function, derived from the definition of mutual information. We evaluate three different policies for choosing the next sensing action: lawn mower, greedy, and non-greedy. For these three policies, we compare the results from our information-theoretic cost functions to the results of other information-theoretic inspired cost functions. Our simulation results show that search efficiency is greater using the proposed cost functions compared to those of the other methods.

Upper and Lower Bounds to the Information Rate Transferred Through First-Order Markov Channels With Free-Running Continuous State

Starting from the definition of mutual information, one promptly realizes that the probabilities inferred by Bayesian tracking can be used to compute the Shannon information between the state and the measurement of a dynamic system. In the Gaussian and linear case, the information rate can be evaluated from the probabilities computed by the Kalman filter. When the probability distributions inferred by Bayesian tracking are nontractable, one is forced to resort to approximated inference, which gives only an approximation to the wanted probabilities. We propose upper and lower bounds to the information rate between the hidden state and the measurement based on approximated inference. Application of these bounds to multiplicative communication channels is discussed, and experimental results for the discrete-time phase noise channel and for the Gauss-Markov fading channel are presented.

Directed Information, Causal Estimation, and Communication in Continuous Time

A notion of directed information between two continuous-time processes is proposed. A key component in the definition is taking an infimum over all possible partitions of the time interval, which plays a role no less significant than the supremum over “space” partitions inherent in the definition of mutual information. Properties and operational interpretations in estimation and communication are then established for the proposed notion of directed information. For the continuous-time additive white Gaussian noise channel, it is shown that Duncan's classical relationship between causal estimation error and mutual information continues to hold in the presence of feedback upon replacing mutual information by directed information. A parallel result is established for the Poisson channel. The utility of this relationship is demonstrated in computing the directed information rate between the input and output processes of a continuous-time Poisson channel with feedback, where the channel input process is constrained to be constant between events at the channel output. Finally, the capacity of a wide class of continuous-time channels with feedback is established via directed information, characterizing the fundamental limit on reliable communication.

Mutual Information Rate and Bounds for It

The amount of information exchanged per unit of time between two nodes in a dynamical network or between two data sets is a powerful concept for analysing complex systems. This quantity, known as the mutual information rate (MIR), is calculated from the mutual information, which is rigorously defined only for random systems. Moreover, the definition of mutual information is based on probabilities of significant events. This work offers a simple alternative way to calculate the MIR in dynamical (deterministic) networks or between two time series (not fully deterministic), and to calculate its upper and lower bounds without having to calculate probabilities, but rather in terms of well known and well defined quantities in dynamical systems. As possible applications of our bounds, we study the relationship between synchronisation and the exchange of information in a system of two coupled maps and in experimental networks of coupled oscillators.

Secrecy Rate of Adaptive Modulation Techniques in Flat-Fading Channels

In this paper, we analyze the Secrecy Rate of Adaptive Modulation transmission systems over flat-fading timevarying channels. The scenario considered consists of a transmitter communicating with a legitimate receiver in the presence of an eavesdropper. Expressions for the Mutual Information (MI) between the transmitter and the legitimate receiver and between the transmitter and the eavesdropper are presented. In addition, the Secrecy Rate is evaluated as the difference between the legitimate receiver MI and the eavesdropper MI. Unlike others studies that are based on a water-filling approach, here the MI expressions are obtained from the classical MI definition considering some assumptions usually adopted in the wireless communication systems context, and they are expressed in terms of important parameters of an adaptive modulation scheme.

Lautum Information

A popular way to measure the degree of dependence between two random objects is by their mutual information, defined as the divergence between the joint and product-of-marginal distributions. We investigate an alternative measure of dependence: the lautum information defined as the divergence between the product-of-marginal and joint distributions, i.e., swapping the arguments in the definition of mutual information. Some operational characterizations and properties are provided for this alternative measure of information.

Blind separation methods based on Pearson system and its extensions

We introduce a mutual information-based method for blind separation of statistically independent source signals. The Pearson system is used as a parametric model. Starting from the definition of mutual information we show using the results by Pham (IEEE Trans. Signal Process. 44(11) (1996) 2768–2779) that the minimization of mutual information contrast leads to iterative use of score functions as estimation functions. The Pearson system allows adaptive modeling of the score functions. The characteristics of the Pearson system are studied and estimators for the parameters are derived using the method of moments. The flexibility of the Pearson system makes it possible to model wide range of source distributions including asymmetric distributions. Skewed source distributions are common in many key application areas, such as telecommunications and biomedical signal processing. We also introduce an extension of the Pearson system that can model multimodal distributions. The applicability of the Pearson system-based method is demonstrated in simulation examples, including blind equalization of GMSK signals.

Generalized cutoff rates and Renyi's information measures

Renyi's (1961) entropy and divergence of order a are given operational characterizations in terms of block coding and hypothesis testing, as so-called /spl beta/-cutoff rates, with /spl alpha/=(1+/spl beta/)/sup -1/ for entropy and /spl alpha/=(1-/spl beta/)/sup -1/ for divergence. Out of several possible definitions of mutual information and channel capacity of order /spl alpha/, our approach distinguishes one that admits an operational characterization as /spl beta/-cutoff rate for channel coding, with /spl alpha/=(1-/spl beta/)/sup -1/. The ordinary cutoff rate of a DMC corresponds to /spl beta/=-1.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Informational Observation Theory: An Extension to Two Continuous Sets of Events

An analysis is proposed of the consequences of the limitations inherent in observation processes PA and PB on the joint, marginal and conditional entropies of two continuous sets A and B of events, observed by means of PA and PB. These events can be statistically dependent. Properties of asymmetry appear which do not exist in the Shannonian model. They induce a new definition of mutual information.

On functionals satisfying a data-processing theorem

It is shown that the rate-distortion bound (R(d) \leq C) remains true when -\log x in the definition of mutual information is replaced by an arbitrary concave (\cup) nonincreasing function satisfying some technical conditions. Examples are given showing that for certain choices of the concave functions, the bounds obtained are better than the classical rate-distortion bounds.