Mutual Information Rate Research Articles

Performance assessment in brain-computer interface-based augmentative and alternative communication.

A large number of incommensurable metrics are currently used to report the performance of brain-computer interfaces (BCI) used for augmentative and alterative communication (AAC). The lack of standard metrics precludes the comparison of different BCI-based AAC systems, hindering rapid growth and development of this technology. This paper presents a review of the metrics that have been used to report performance of BCIs used for AAC from January 2005 to January 2012. We distinguish between Level 1 metrics used to report performance at the output of the BCI Control Module, which translates brain signals into logical control output, and Level 2 metrics at the Selection Enhancement Module, which translates logical control to semantic control. We recommend that: (1) the commensurate metrics Mutual Information or Information Transfer Rate (ITR) be used to report Level 1 BCI performance, as these metrics represent information throughput, which is of interest in BCIs for AAC; 2) the BCI-Utility metric be used to report Level 2 BCI performance, as it is capable of handling all current methods of improving BCI performance; (3) these metrics should be supplemented by information specific to each unique BCI configuration; and (4) studies involving Selection Enhancement Modules should report performance at both Level 1 and Level 2 in the BCI system. Following these recommendations will enable efficient comparison between both BCI Control and Selection Enhancement Modules, accelerating research and development of BCI-based AAC systems.

Symbolic transfer entropy rate is equal to transfer entropy rate for bivariate finite-alphabet stationary ergodic Markov processes

Transfer entropy is a measure of the magnitude and the direction of information flow between jointly distributed stochastic processes. In recent years, its permutation analogues are considered in the literature to estimate the transfer entropy by counting the number of occurrences of orderings of values, not the values themselves. It has been suggested that the method of permutation is easy to implement, computationally low cost and robust to noise when applying to real world time series data. In this paper, we initiate a theoretical treatment of the corresponding rates. In particular, we consider the transfer entropy rate and its permutation analogue, the symbolic transfer entropy rate, and show that they are equal for any bivariate finite-alphabet stationary ergodic Markov process. This result is an illustration of the duality method introduced in [T. Haruna, K. Nakajima, Physica D 240, 1370 (2011)]. We also discuss the relationship among the transfer entropy rate, the time-delayed mutual information rate and their permutation analogues.

Information-theoretic versus thermodynamic entropy production in autonomous sensory networks

For sensory networks, we determine the rate with which they acquire information about the changing external conditions. Comparing this rate with the thermodynamic entropy production that quantifies the cost of maintaining the network, we find that there is no universal bound restricting the rate of obtaining information to be less than this thermodynamic cost. These results are obtained within a general bipartite model consisting of a stochastically changing environment that affects the instantaneous transition rates within the system. Moreover, they are illustrated with a simple four-states model motivated by cellular sensing. On the technical level, we obtain an upper bound on the rate of mutual information analytically and calculate this rate with a numerical method that estimates the entropy of a time series generated with a simulation.

Information, Estimation, and Lookahead in the Gaussian Channel

We consider mean squared estimation with lookahead of a continuous-time signal corrupted by additive white Gaussian noise. We show that the mutual information rate function, i.e., the mutual information rate as function of the signal-to-noise ratio (SNR), does not, in general, determine the minimum mean squared error (MMSE) with fixed finite lookahead, in contrast to the special cases with 0 and infinite lookahead (filtering and smoothing errors), respectively, which were previously established in the literature. We also establish a new expectation identity under a generalized observation model where the Gaussian channel has an SNR jump at $t=0$, capturing the tradeoff between lookahead and SNR. Further, we study the class of continuous-time stationary Gauss-Markov processes (Ornstein-Uhlenbeck processes) as channel inputs, and explicitly characterize the behavior of the minimum mean squared error (MMSE) with finite lookahead and signal-to-noise ratio (SNR). The MMSE with lookahead is shown to converge exponentially rapidly to the non-causal error, with the exponent being the reciprocal of the non-causal error. We extend our results to mixtures of Ornstein-Uhlenbeck processes, and use the insight gained to present lower and upper bounds on the MMSE with lookahead for a class of stationary Gaussian input processes, whose spectrum can be expressed as a mixture of Ornstein-Uhlenbeck spectra.

MIMO Exploitation of 3D Multipath Statistics in a Heterogeneous LTE-Advanced Network

This paper analyses the impact of 3D multipath in an LTE-Advanced heterogeneous network. The impact of the base station array configuration is considered in both the azimuth and elevation planes. Spatial and temporal multipath statistics are generated for example macro- and picocellular base stations, and these are combined with appropriately oriented complex polarimetric antenna patterns. The resulting wideband channels are then passed to an LTE-Advanced physical layer simulator. The optimal spatial multiplexing mode is determined by computing the throughput using the received bit mutual information rate for all modulation and coding schemes. The fastest link speed for each user is identified given a packet error rate threshold of 10%. Results show that RMS angle spread statistics and the base station MIMO array configuration strongly influence user performance in an LTE-A network. The deployment of macrocellular vertical arrays is shown to significantly degrade network performance. In contrast, the elevation angle spread is found to be far greater in picocells, and this allows vertical arrays to be applied to produce compact high performance picobase stations.

Mutual Information Rate and Bounds for It

The amount of information exchanged per unit of time between two nodes in a dynamical network or between two data sets is a powerful concept for analysing complex systems. This quantity, known as the mutual information rate (MIR), is calculated from the mutual information, which is rigorously defined only for random systems. Moreover, the definition of mutual information is based on probabilities of significant events. This work offers a simple alternative way to calculate the MIR in dynamical (deterministic) networks or between two time series (not fully deterministic), and to calculate its upper and lower bounds without having to calculate probabilities, but rather in terms of well known and well defined quantities in dynamical systems. As possible applications of our bounds, we study the relationship between synchronisation and the exchange of information in a system of two coupled maps and in experimental networks of coupled oscillators.

CSI Usage over Parallel Fading Channels under Jamming Attacks: A Game Theory Study

Consider a parallel channel with M independent flat-fading subchannels. There exists a smart jammer which has possession of a copy of perfect channel state information (CSI) measured and sent back by a receiver to its transmitter. Under this model, a class of two-person zero-sum games is investigated where either achievable mutual information rate or Chernoff bound is taken as the underlying pay-off function with the strategy space of each player determined by respective power control and hopping functions. More specifically, we have tackled and answered the following three fundamental questions. The first one is about whether the transmitter and jammer should hop or fully use all degrees of freedom over the entire parallel channels given the full CSI available to both of them, i.e. to hop or not to hop. The second question is about the impact of sending back CSI on system performance considering that the smart jammer can exploit CSI to further enhance its interference effects, i.e. to feedback or not to feedback. The last question is about whether the amount of feedback information can be reduced given the mutual restrictions between transmitter and jammer, i.e. when to feedback and when not to.

Generalized Belief Propagation for the Noiseless Capacity and Information Rates of Run-Length Limited Constraints

The performance of the generalized belief propagation algorithm to compute the noiseless capacity and mutual information rates of finite-size two-dimensional and three-dimensional run-length limited constraints is investigated. In both cases, the problem is reduced to estimating the partition function of graphical models with cycles. The partition function is then estimated using the region-based free energy approximation technique. For each constraint, a method is proposed to choose the basic regions and to construct the region graph which provides the graphical framework to run the generalized belief propagation algorithm. Simulation results for the noiseless capacity of different constraints as a function of the size of the channel are reported. In the cases that tight lower and upper bounds on the Shannon capacity exist, convergence to the Shannon capacity is discussed. For noisy constrained channels, simulation results are reported for mutual information rates as a function of signal-to-noise ratio.

Concavity of the Mutual Information Rate for Input-Restricted Memoryless Channels at High SNR

We consider a memoryless channel with an input Markov process supported on a mixing finite-type constraint. We continue the development of asymptotics for the entropy rate of the output hidden Markov chain and deduce that, at high signal-to-noise ratio, the mutual information rate of such a channel is concave with respect to “almost” all input Markov chains of a given order.

Transmit Diversity and Relay Selection Algorithms for Multirelay Cooperative MIMO Systems

In this paper, we propose a set of joint transmit diversity selection (TDS) and relay selection (RS) algorithms based on discrete iterative stochastic optimization for the uplink of cooperative multiple-input-multiple-output (MIMO) systems. Decode-and-forward (DF) and amplify-and-forward (AF) multirelay systems with linear minimum mean square error (MSE), successive interference cancelation, and adaptive reception are considered. The problems of TDS and RS are expressed as MSE and mutual information (MI) joint discrete optimization problems and solved using iterative discrete stochastic algorithms. Such an approach circumvents the need for exhaustive searching and results in a range of procedures with low complexity and increased speed of convergence that can track the optimal selection over an estimated channel. The proposed schemes are analyzed in terms of their complexity, convergence, and diversity benefits and are shown to be both stable and computationally efficient. Their performance is then evaluated via MSE, MI, and bit error rate comparisons and shown to outperform conventional cooperative transmission and, in the majority of scenarios, match that of the optimal exhaustive solution.

Relay Network Beamforming and Power Control Using Maximization of Mutual Information

In this paper, distributed beamforming and power control are considered for amplify-and-forward wireless relay networks with single antenna nodes. The optimal relay weights are designed by maximizing the mutual information in the case of perfect channel state information, individual node power constraints and the existence of a direct link between the source and destination nodes. In our approach, the source node transmits up to two precoded independent information streams as opposed to several previously proposed techniques in which a single symbol is always transmitted. Our solution to the relay power control given the power allocation at the source node is obtained analytically with linear complexity. Simulations show that the proposed method outperforms the previously proposed techniques in terms of mutual information, outage probability and bit error rate.

On the spectral formulation of Granger causality

Spectral measures of causality are used to explore the role of different rhythms in the causal connectivity between brain regions. We study several spectral measures related to Granger causality, comprising the bivariate and conditional Geweke measures, the directed transfer function, and the partial directed coherence. We derive the formulation of dependence and causality in the spectral domain from the more general formulation in the information-theory framework. We argue that the transfer entropy, the most general measure derived from the concept of Granger causality, lacks a spectral representation in terms of only the processes associated with the recorded signals. For all the spectral measures we show how they are related to mutual information rates when explicitly considering the parametric autoregressive representation of the processes. In this way we express the conditional Geweke spectral measure in terms of a multiple coherence involving innovation variables inherent to the autoregressive representation. We also link partial directed coherence with Sims' criterion of causality. Given our results, we discuss the causal interpretation of the spectral measures related to Granger causality and stress the necessity to explicitly consider their specific formulation based on modeling the signals as linear Gaussian stationary autoregressive processes.

Delay independence of mutual-information rate of two symbolic sequences

Introduced by Shannon as a "rate of actual transmission," mutual information rate (MIR) is an extension of mutual information to a pair of dynamical processes. We show a delay-independence theorem, according to which MIR is not sensitive to a time shift between the two processes. Numerical studies of several benchmark situations confirm that this theoretical asymptotic property remains valid for realistic finite sequences. Estimations based on block entropies and a causal state machine algorithm perform better than an estimation based on a Lempel-Ziv compression algorithm provided that block length and maximum history length, respectively, can be chosen larger than the delay. MIR is thus a relevant index for measuring nonlinear correlations between two experimental or simulated sequences when the transmission delay (in input-output devices) or dephasing (in coupled systems) is variable or unknown.

BIT-TRAPS: Building Information-Theoretic Traffic Privacy Into Packet Streams

Sniffing encrypted data packets traveling across networks can often be useful in inferring nontrivial information about their contents because of the manner in which the transmission of such packets is handled by lower layers in the communications protocol stack. In this paper, we formally study the side-channel formed by variable packet sizes, and explore obfuscation approaches to prevent information leakage while jointly considering the practical cost of obfuscation. We show that randomized algorithms for obfuscation perform best and can be studied as well-known information-theoretic constructs, such as discrete channels with and without memory. We envision a separate layer called a Bit - Trap, that employs buffering and bit-padding as orthogonal methods for obfuscating such side channels. For streams of packets, we introduce the use of mutual-information rate as an appropriate metric for the level of obfuscation that captures nonlinear relationships between original and modified streams. Using buffering-delay and average bit-padding as the respective costs, a Bit - Trap formulates a constrained optimization problem with bounds on the average costs, to implement the best possible obfuscation policy. We find that combining small amounts of delay and padding together can create much more obfuscation than either approach alone, and that a simple convex trade-off exists between buffering delay and padding for a given level of obfuscation.

Information analysis of posterior canal afferents in the turtle, Trachemys scripta elegans

We have used sinusoidal and band-limited Gaussian noise stimuli along with information measures to characterize the linear and non-linear responses of morpho-physiologically identified posterior canal (PC) afferents and to examine the relationship between mutual information rate and other physiological parameters. Our major findings are: 1) spike generation in most PC afferents is effectively a stochastic renewal process, and spontaneous discharges are fully characterized by their first order statistics; 2) a regular discharge, as measured by normalized coefficient of variation (cv*), reduces intrinsic noise in afferent discharges at frequencies below the mean firing rate; 3) coherence and mutual information rates, calculated from responses to band-limited Gaussian noise, are jointly determined by gain and intrinsic noise (discharge regularity), the two major determinants of signal to noise ratio in the afferent response; 4) measures of optimal non-linear encoding were only moderately greater than optimal linear encoding, indicating that linear stimulus encoding is limited primarily by internal noise rather than by non-linearities; and 5) a leaky integrate and fire model reproduces these results and supports the suggestion that the combination of high discharge regularity and high discharge rates serves to extend the linear encoding range of afferents to higher frequencies. These results provide a framework for future assessments of afferent encoding of signals generated during natural head movements and for comparison with coding strategies used by other sensory systems.This article is part of a Special Issue entitled: Neural Coding.

A mutual information‐based metric for evaluation of fMRI data‐processing approaches

We propose a novel approach for evaluating the performance of activation detection in real (experimental) datasets using a new mutual information (MI)-based metric and compare its sensitivity to several existing performance metrics in both simulated and real datasets. The proposed approach is based on measuring the approximate MI between the fMRI time-series of a validation dataset and a calculated activation map (thresholded label map or continuous map) from an independent training dataset. The MI metric is used to measure the amount of information preserved during the extraction of an activation map from experimentally related fMRI time-series. The processing method that preserves maximal information between the maps and related time-series is proposed to be superior. The results on simulation datasets for multiple analysis models are consistent with the results of ROC curves, but are shown to have lower information content than for real datasets, limiting their generalizability. In real datasets for group analyses using the general linear model (GLM; FSL4 and SPM5), we show that MI values are (1) larger for groups of 15 versus 10 subjects and (2) more sensitive measures than reproducibility (for continuous maps) or Jaccard overlap metrics (for thresholded maps). We also show that (1) for an increasing fraction of nominally active voxels, both MI and false discovery rate (FDR) increase, and (2) at a fixed FDR, GLM using FSL4 tends to extract more voxels and more information than SPM5 using the default processing techniques in each package.

Capacity-achieving probability measure for a reduced number of signaling points

Putting bounding constraints on the input of a channel leads in many cases to a discrete capacity-achieving distribution with a finite support. Given a finite number of signaling points, we determine reduced subsets and the corresponding optimal probability measures to simplify the receiver design. The objective for the subset selection is to keep the channel quality high by maximizing mutual information and cutoff rate. Two approaches are introduced to obtain a capacity-achieving probability measure for the reduced subset. The first one is based on a preceded signaling point selection while the second one chooses the signaling points and corresponding probabilities simultaneously. Numerical results for both approaches show that using only a small number of signaling points achieves a very high mutual information compared to channels utilizing the full set of signaling points.

Brain-Generated Estradiol Drives Long-Term Optimization of Auditory Coding to Enhance the Discrimination of Communication Signals

Auditory processing and hearing-related pathologies are heavily influenced by steroid hormones in a variety of vertebrate species, including humans. The hormone estradiol has been recently shown to directly modulate the gain of central auditory neurons, in real time, by controlling the strength of inhibitory transmission via a nongenomic mechanism. The functional relevance of this modulation, however, remains unknown. Here we show that estradiol generated in the songbird homolog of the mammalian auditory association cortex, rapidly enhances the effectiveness of the neural coding of complex, learned acoustic signals in awake zebra finches. Specifically, estradiol increases mutual information rates, coding efficiency, and the neural discrimination of songs. These effects are mediated by estradiol's modulation of both the rate and temporal coding of auditory signals. Interference with the local action or production of estradiol in the auditory forebrain of freely behaving animals disrupts behavioral responses to songs, but not to other behaviorally relevant communication signals. Our findings directly show that estradiol is a key regulator of auditory function in the adult vertebrate brain.

Inhibition of SK and M channel-mediated currents by 5-HT enables parallel processing by bursts and isolated spikes.

Although serotonergic innervation of sensory brain areas is ubiquitous, its effects on sensory information processing remain poorly understood. We investigated these effects in pyramidal neurons within the electrosensory lateral line lobe (ELL) of weakly electric fish. Surprisingly, we found that 5-HT is present at different levels across the different ELL maps; the presence of 5-HT fibers was highest in the map that processes intraspecies communication signals. Electrophysiological recordings revealed that 5-HT increased excitability and burst firing through a decreased medium afterhyperpolarization resulting from reduced small-conductance calcium-activated (SK) currents as well as currents mediated by an M-type potassium channel. We next investigated how 5-HT alters responses to sensory input. 5-HT application decreased the rheobase current, increased the gain, and decreased first spike latency. Moreover, it reduced discriminability between different stimuli, as quantified by the mutual information rate. We hypothesized that 5-HT shifts pyramidal neurons into a burst-firing mode where bursts, when considered as events, can detect the presence of particular stimulus features. We verified this hypothesis using signal detection theory. Our results indeed show that serotonin-induced bursts of action potentials, when considered as events, could detect specific stimulus features that were distinct from those detected by isolated spikes. Moreover, we show the novel result that isolated spikes transmit more information after 5-HT application. Our results suggest a novel function for 5-HT in that it enables differential processing by action potential patterns in response to current injection.

Information theoretic interpretation of frequency domain connectivity measures

In order to provide adequate multivariate measures of information flow between neural structures, modified expressions of partial directed coherence (PDC) and directed transfer function (DTF), two popular multivariate connectivity measures employed in neuroscience, are introduced and their formal relationship to mutual information rates are proved.