Mutual Information Rate Research Articles

Coding of Stimuli by Ampullary Afferents inGnathonemus petersii

Weakly electric fish use electroreception for both active and passive electrolocation and for electrocommunication. While both active and passive electrolocation systems are prominent in weakly electric Mormyriform fishes, knowledge of their passive electrolocation ability is still scarce. To better estimate the contribution of passive electric sensing to the orientation toward electric stimuli in weakly electric fishes, we investigated frequency tuning applying classical input-output characterization and stimulus reconstruction methods to reveal the encoding capabilities of ampullary receptor afferents. Ampullary receptor afferents were most sensitive (threshold: 40 μV/cm) at low frequencies (<10 Hz) and appear to be tuned to a mix of amplitude and slope of the input signals. The low-frequency tuning was corroborated by behavioral experiments, but behavioral thresholds were one order of magnitude higher. The integration of simultaneously recorded afferents of similar frequency-tuning resulted in strongly enhanced signal-to-noise ratios and increased mutual information rates but did not increase the range of frequencies detectable by the system. Theoretically the neuronal integration of input from receptors experiencing opposite polarities of a stimulus (left and right side of the fish) was shown to enhance encoding of such stimuli, including an increase of bandwidth. Covariance and coherence analysis showed that spiking of ampullary afferents is sufficiently explained by the spike-triggered average, i.e., receptors respond to a single linear feature of the stimulus. Our data support the notion of a division of labor of the active and passive electrosensory systems in weakly electric fishes based on frequency tuning. Future experiments will address the role of central convergence of ampullary input that we expect to lead to higher sensitivity and encoding power of the system.

Limitations in tracking systems

In this paper, we obtain information theoretical conditions for tracking in linear time-invariant control systems. We consider the particular case where the closed loop contains a channel in the feedback loop. The mutual information rate between the feedback signal and the reference input signal is used to quantify information about the reference signal that is available for feedback. This mutual information rate must be maximized in order to improve the tracking performance. The mutual information is shown to be upper bounded by a quantity that depends on the unstable eigenvalues of the plant and on the channel capacity. If the channel capacity reaches a lower limit, the feedback signal becomes completely uncorrelated with the reference signal, rendering feedback useless. We also find a lower bound on the expected squared tracking error in terms of the entropy of a random reference signal. We show a misleading case where the mutual information rate does not predict the expected effect of nonminimum phase zeros. However, mutual information rate helps generalize the concept that there is a tradeoff when tracking and disturbance rejection are simultaneous goals, and a constraint communication channel is present in the feedback loop. Examples and simulations are provided to demonstrate some of the results.

Enhancement of information transmission of sub-threshold signals applied to distal positions of dendritic trees in hippocampal CA1 neuron models with stochastic resonance

Stochastic resonance (SR) has been shown to enhance the signal-to-noise ratio and detection of low level signals in neurons. It is not yet clear how this effect of SR plays an important role in the information processing of neural networks. The objective of this article is to test the hypothesis that information transmission can be enhanced with SR when sub-threshold signals are applied to distal positions of the dendrites of hippocampal CA1 neuron models. In the computer simulation, random sub-threshold signals were presented repeatedly to a distal position of the main apical branch, while the homogeneous Poisson shot noise was applied as a background noise to the mid-point of a basal dendrite in the CA1 neuron model consisting of the soma with one sodium, one calcium, and five potassium channels. From spike firing times recorded at the soma, the mutual information and information rate of the spike trains were estimated. The simulation results obtained showed a typical resonance curve of SR, and that as the activity (intensity) of sub-threshold signals increased, the maximum value of the information rate tended to increased and eventually SR disappeared. It is concluded that SR can play a key role in enhancing the information transmission of sub-threshold stimuli applied to distal positions on the dendritic trees.

Efficient Heuristics for Discriminative Structure Learning of Bayesian Network Classifiers

We introduce a simple order-based greedy heuristic for learning discriminative structure within generative Bayesian network classifiers. We propose two methods for establishing an order of N features. They are based on the conditional mutual information and classification rate (i.e., risk), respectively. Given an ordering, we can find a discriminative structure with O(Nk+1) score evaluations (where constant k is the tree-width of the sub-graph over the attributes). We present results on 25 data sets from the UCI repository, for phonetic classification using the TIMIT database, for a visual surface inspection task, and for two handwritten digit recognition tasks. We provide classification performance for both discriminative and generative parameter learning on both discriminatively and generatively structured networks. The discriminative structure found by our new procedures significantly outperforms generatively produced structures, and achieves a classification accuracy on par with the best discriminative (greedy) Bayesian network learning approach, but does so with a factor of ~10-40 speedup. We also show that the advantages of generative discriminatively structured Bayesian network classifiers still hold in the case of missing features, a case where generative classifiers have an advantage over discriminative classifiers.

Information Theoretic Bounds for Compound MIMO Gaussian Channels

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> In this paper, achievable rates for compound Gaussian multiple-input–multiple-output (MIMO) channels are derived. Two types of channels, modeled in the frequency domain, are considered when: 1) the channel frequency response matrix <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$H$</tex> </formula></emphasis> belongs to a subset of <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$H^{\infty}$</tex></formula></emphasis> normed linear space, and 2) the power spectral density (PSD) matrix of the Gaussian noise belongs to a subset of <emphasis emphasistype="italic"><formula formulatype="inline"> <tex Notation="TeX">$L_1$</tex></formula></emphasis> space. The achievable rates of these two compound channels are related to the maximin of the mutual information rate. The minimum is with respect to the set of all possible <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$H$</tex> </formula></emphasis> matrices or all possible PSD matrices of the noise. The maximum is with respect to all possible PSD matrices of the transmitted signal with bounded power. For the compound channel modeled by the set of <emphasis emphasistype="italic"><formula formulatype="inline"><tex Notation="TeX">$H$</tex> </formula></emphasis> matrices, it is shown, under certain conditions, that the code for the worst case channel can be used for the whole class of channels. For the same model, the water-filling argument implies that the larger the set of matrices <emphasis emphasistype="italic"><formula formulatype="inline"> <tex Notation="TeX">$H$</tex></formula></emphasis>, the smaller the bandwidth of the transmitted signal will be. For the second compound channel, the explicit relation between the maximizing PSD matrix of the transmitted signal and the minimizing PSD matrix of the noise is found. Two PSD matrices are related through a Riccati equation, which is always present in Kalman filtering and liner-quadratic Gaussian control problems. </para>

On information transmission in linear feedback tracking systems

AbstractInformation transmission in discrete time linear time‐invariant (LTI) feedback tracking systems was investigated by using measures of directed information and mutual information. It was proved that, for a pair of extraneous input and internal variable, directed information (rate) is always equal to mutual information (rate); for a pair of internal variables, the former is smaller than the latter. Furthermore, the feedback changes the information transmission between internal variables, while it has no influence on information transmission from extraneous variable to internal variable. Consideration on system design was discussed. Copyright © 2008 Curtin University of Technology and John Wiley &amp; Sons, Ltd.

Robust wavelet estimation and blind deconvolution of noisy surface seismics

Robust blind deconvolution is a challenging problem, particularly if the bandwidth of the seismic wavelet is narrow to very narrow; that is, if the wavelet bandwidth is similar to its principal frequency. The main problem is to estimate the phase of the wavelet with sufficient accuracy. The mutual information rate is a general-purpose criterion to measure whiteness using statistics of all orders. We modified this criterion to measure robustly the amplitude and phase spectrum of the wavelet in the presence of noise. No minimum phase assumptions were made. After wavelet estimation, we obtained an optimal deconvolution output using Wiener filtering. The new procedure performs well, even for very band-limited data; and it produces frequency-dependent phase estimates.

Network mutual information and synchronization under time transformations

We investigate the effect of general time transformations on the phase synchronization (PS) phenomenon and the mutual information rate (MIR) between pairs of nodes in dynamical networks. We demonstrate two important results concerning the invariance of both PS and the MIR. Under time transformations PS can neither be introduced nor destroyed and the MIR cannot be raised from zero. On the other hand, for proper time transformations the timing between the cycles of the coupled oscillators can be largely improved. Finally, we discuss the relevance of our findings for communication in dynamical networks.

Analog Precoder and Equalizer Designs and their Geometry for Multichannel Communication

This is a paper on modulation theory that addresses joint analog precoder and equalizer design for multichannel data transmission over the frequency-selective additive Gaussian noise (AGN) channel. The design goal is to maximize mutual information rate, minimize the mean square error, or minimize the bit error rate subject to a transmit power constraint. We assume a continuous channel model with precoder transmissions for m subchannels that lie in an n-dimensional linear subspace of L <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> (R). m and n are design parameters. We first design the subspace according to the channel characteristics, and then design the precoders as functions in this subspace. After the design of the optimal precoder and equalizer, we explore the geometry of these designs. We show that all of these precoder and equalizer designs are, in fact, decompositions of a virtual two- channel problem into a system of canonical coordinates, wherein variables in the canonical message channel are correlated only pairwise with corresponding variables in the canonical measurement channel. This finding clarifies the geometry of precoder and equalizer designs and illustrates that they decompose the two-channel communication problem into what might be called the Shannon channel.

Optimization of CDMA Systems with Respect to Transmission Probability, Part I: Mutual Information Rate Optimization

Direct sequence code division multiple access (CDMA) systems that use non-continuous transmission were considered throughout the history of spread spectrum systems, but gained renewed interest with the emergence of impulse radio (IR) technology. Recently, several works had shown that in non- continuous CDMA, the transmission duty cycle (or transmission probability) has a significant effect on system performance. In this work we address the optimization of the performance of CDMA systems with adjustable transmission probabilities. We consider CDMA systems that implement non-continuous transmission by random puncturing, and study the system optimization with respect to both transmission powers (termed power control) and transmission probabilities (termed probability control). We show that the joint optimization has a significant performance advantage over the optimization with respect to transmissions powers only. For some cases we even show that the optimization with respect to transmission probabilities alone is sufficient to achieve optimal performance. In this part, we demonstrate the importance of probability control, by studying the case of frequency-flat slow-fading multiple access channel (MAC) and no spreading. We prove, for this special case, that mutual information rate optimization is achieved by probability control only, while power control is redundant. The theory is supported by simulation results, which show that the achievable rates of all users are better in a system that uses probability control instead of power control.

Optimization of CDMA Systems with Respect to Transmission Probability, Part II: Signal to Noise Plus Interference Ratio Optimization

Code division multiple access (CDMA) systems commonly use power control mechanism to reduce the amount of interference between the users. In part I of this paper, we introduced the concept of probability control that optimizes system performance with respect to the users' transmission probabilities. The importance of probability control was demonstrated by proving that probability control alone optimizes the mutual information rates over a frequency-flat slow-fading multiple access channel. In this part we extend this result to the optimization of the average signal-to-noise plus interference ratio (SIR) in a CDMA system over a frequency-selective slow- fading channel model and any network topology. We prove that there is a group of CDMA receivers, in which the average SIR (ASIR) of all users is maximized using probability control (while all users transmit their maximal allowed power). This receivers group includes, among others, the common matched filter (MF) RAKE receiver, as well as the sophisticated minimal mean square error (MMSE) RAKE receiver. Simulations demonstrate for a 2- users scenario that both users achieve significantly higher ASIR using probability-control instead of power-control. Results are applicable both for CDMA and impulse radio (IR) systems.

A Generalization of the Blahut–Arimoto Algorithm to Finite-State Channels

The classical Blahut-Arimoto algorithm (BAA) is a well-known algorithm that optimizes a discrete memoryless source (DMS) at the input of a discrete memoryless channel (DMC) in order to maximize the mutual information between channel input and output. This paper considers the problem of optimizing finite-state machine sources (FSMSs) at the input of finite-state machine channels (FSMCs) in order to maximize the mutual information rate between channel input and output. Our main result is an algorithm that efficiently solves this problem numerically; thus, we call the proposed procedure the generalized BAA. It includes as special cases not only the classical BAA but also an algorithm that solves the problem of finding the capacity-achieving input distribution for finite-state channels with no noise. While we present theorems that characterize the local behavior of the generalized BAA, there are still open questions concerning its global behavior; these open questions are addressed by some conjectures at the end of the paper. Apart from these algorithmic issues, our results lead to insights regarding the local conditions that the information-rate-maximizing FSMSs fulfill; these observations naturally generalize the well-known Kuhn-Tucker conditions that are fulfilled by capacity-achieving DMSs at the input of DMCs.

Basic binary state-space model for time-varying communication channels: uncertainty and information capacity

This paper looks at capacity achieving detection strategies for information transfer over time-varying channels. The time-varying binary symmetric channel (TV-BSC) is identified as the basic binary state-space model. Separation of entropies principles and the TV-BSC model-based state-space approach are used to determine the performance bounds for coherent and non-coherent detection over time-varying communication channels. The mutual information rate over the TV-BSC, assuming channel estimation in the presence of channel noise, is shown to be below the channel information capacity because of lack of perfect channel knowledge. Furthermore, it is shown that TV-BSC model-based differential detection has a fundamental advantage over the channel estimation based detection since it theoretically preserves the TV-BSC information capacity when the observation interval approaches infinity. Simulation analysis corroborates the theoretical results, showing that multiple-symbol differential detection practically achieves the TV-BSC capacity in just a few symbol observation times.

Information transfer rate in fMRI experiments measured using mutual information theory

Information theory provides a mathematical framework for analysis of fMRI experiments. By modeling the fMRI experiment as a communication system, various results from information theory can be applied to measure information transfer rate in fMRI experiments. The information transfer rate has important implications for design and analysis of brain–computer interface (BCI) experiments. A key factor in the effective implementation of BCI techniques is to achieve maximum information transfer rate. In this report, mutual information rate (MIR) was used to evaluate the efficiency of alternative experimental designs. The channel capacity, a fundamental physical limit on the rate at which information can be extracted from an fMRI experiment, was estimated and compared with the theoretical limit specified by the Hartley–Shannon Theorem. We present an information theory framework for the analysis of fMRI time-series assuming a known hemodynamic response function. Using MIR to evaluate fMRI experimental designs, we show that block lengths of 3–5 s have maximum information transfer rates. For designs with shorter block lengths, the MIR is limited by the channel capacity. For experimental designs with longer block lengths, the MIR is limited by the low source information transmission rate.

Estimating Information Rates with Confidence Intervals in Neural Spike Trains

Information theory provides a natural set of statistics to quantify the amount of knowledge a neuron conveys about a stimulus. A related work (Kennel, Shlens, Abarbanel, & Chichilnisky, 2005) demonstrated how to reliably estimate, with a Bayesian confidence interval, the entropy rate from a discrete, observed time series. We extend this method to measure the rate of novel information that a neural spike train encodes about a stimulus--the average and specific mutual information rates. Our estimator makes few assumptions about the underlying neural dynamics, shows excellent performance in experimentally relevant regimes, and uniquely provides confidence intervals bounding the range of information rates compatible with the observed spike train. We validate this estimator with simulations of spike trains and highlight how stimulus parameters affect its convergence in bias and variance. Finally, we apply these ideas to a recording from a guinea pig retinal ganglion cell and compare results to a simple linear decoder.

Determining and Approaching Achievable Rates of Binary Intersymbol Interference Channels Using Multistage Decoding

By examining the achievable rates of a multistage decoding system on stationary ergodic channels, we derive lower bounds on the mutual information rate corresponding to independent and uniformly distributed (i.u.d.) inputs, also referred to as the i.u.d. information rate. For binary intersymbol interference (ISI) channels, we show that these bounds become tight as the number of decoding stages increases. Our analysis, which focuses on the marginal conditional output densities at each stage of decoding, provides an information rate corresponding to each stage. These rates underlie the design of multilevel coding schemes, based upon low-density parity-check (LDPC) codes and message passing, that in combination with multistage decoding approach the i.u.d. information rate for binary ISI channels. We give example constructions for channel models that have been commonly used in magnetic recording. These examples demonstrate that the technique is very effective even for a small number of decoding stages

Interval Coding. II. Dendrite-Dependent Mechanisms

The rich temporal structure of neural spike trains provides multiple dimensions to code dynamic stimuli. Popular examples are spike trains from sensory cells where bursts and isolated spikes can serve distinct coding roles. In contrast to analyses of neural coding, the cellular mechanics of burst mechanisms are typically elucidated from the neural response to static input. Bridging the mechanics of bursting with coding of dynamic stimuli is an important step in establishing theories of neural coding. Electrosensory lateral line lobe (ELL) pyramidal neurons respond to static inputs with a complex dendrite-dependent burst mechanism. Here we show that in response to dynamic broadband stimuli, these bursts lack some of the electrophysiological characteristics observed in response to static inputs. A simple leaky integrate-and-fire (LIF)-style model with a dendrite-dependent depolarizing afterpotential (DAP) is sufficient to match both the output statistics and coding performance of experimental spike trains. We use this model to investigate a simplification of interval coding where the burst interspike interval (ISI) codes for the scale of a canonical upstroke rather than a multidimensional stimulus feature. Using this stimulus reduction, we compute a quantization of the burst ISIs and the upstroke scale to show that the mutual information rate of the interval code is maximized at a moderate DAP amplitude. The combination of a reduced description of ELL pyramidal cell bursting and a simplification of the interval code increases the generality of ELL burst codes to other sensory modalities.

Frequency-domain blind deconvolution based on mutual information rate

In this paper, a new blind single-input single-output (SISO) deconvolution method based on the minimization of the mutual information rate of the deconvolved output is proposed. The method works in the frequency domain and requires estimation of the signal probability density function. Thus, the algorithm uses higher order statistics (except for Gaussian source) and allows non-minimum-phase filter estimation. In practice, the criterion contains a regularization term for limiting noise amplification as in Wiener filtering. The score function estimation, which represents a key point of the algorithm, is detailed, and the most robust estimate is selected. Finally, experiments point to the relevance of the proposed algorithm: 1) any filter, minimum phase or not, can be estimated and 2) on actual data (underwater explosions, seismovolcanic phenomena), this deconvolution algorithm provides good results with a better tradeoff between deconvolution quality and noise amplification than existing methods.

Capacity of MIMO Rician channels

This paper presents exact results on the capacity of multiple-input-multiple-output (MIMO) Rician channels when perfect channel state information (CSI) is assumed at the receiver but the transmitter has neither instantaneous nor statistical CSI. It first derives the exact expression for the average mutual information (MI) rate of MIMO Rician fading channels when the fading coefficients are independent but not necessarily identically distributed. The results for the independent and identically distributed (i.i.d.) MIMO Rician and Rayleigh fading channels are also obtained as special cases. These results are derived using a different approach than the one used by Telatar for the i.i.d. Rayleigh case. The complementary cumulative distribution function (CCDF) of the MI is also obtained using a Gaussian approximation. The CDF of MI can serve as an upper bound to the outage probability of nonergodic MIMO Rician channels. Numerical results confirm that for a fixed channel gain, a strong tine-of-sight component decreases the channel capacity due to the lack of scattering.

Low-complexity bit-interleaved coded DAPSK for Rayleigh-fading channels

We analyze the achievable performance of bit-interleaved coded differential amplitude and phase-shift keying (DAPSK) systems over frequency nonselective Rayleigh-fading channels with suboptimal differential detection assuming an ideal bit interleaving. The suboptimal differential detection in this work refers to the bit metric calculation based only on the difference between two consecutive symbols, in contrast to more complex maximum-likelihood (ML)-based differential detection, which makes use of all the observed consecutive symbols for its metric calculation in channel decoding. As benchmarks of coded system performance, we analyze the average mutual information (AMI) and cutoff rate of this system. Exact probability density functions of the suboptimal differential detector outputs are derived for this purpose. Comparative studies suggest that the performance loss of the suboptimal approach is in fact noticeable. Therefore, we also develop a low-complexity receiver structure in the framework of suboptimal differential detection that can approach the performance of ML-based system by suitably incorporating the amplitude statistics of received symbols. The theoretical framework developed in this paper is also confirmed by simulations using convolutional and turbo codes.